Inhomogeneous behaviour for descriptors in between the class-instance and metaclass-class pairs

[HugoRicateau]

BPO	39443
Nosy	@rhettinger, @ericsnowcurrently

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/rhettinger'
closed_at = <Date 2020-01-26.06:04:03.108>
created_at = <Date 2020-01-24.15:17:34.433>
labels = ['invalid', 'type-bug', '3.8', 'docs']
title = 'Inhomogeneous behaviour for descriptors in between the class-instance and metaclass-class pairs'
updated_at = <Date 2020-01-26.06:04:03.107>
user = 'https://bugs.python.org/HugoRicateau'

bugs.python.org fields:

activity = <Date 2020-01-26.06:04:03.107>
actor = 'rhettinger'
assignee = 'rhettinger'
closed = True
closed_date = <Date 2020-01-26.06:04:03.108>
closer = 'rhettinger'
components = ['Documentation']
creation = <Date 2020-01-24.15:17:34.433>
creator = 'Hugo Ricateau'
dependencies = []
files = []
hgrepos = []
issue_num = 39443
keywords = []
message_count = 5.0
messages = ['360623', '360644', '360645', '360685', '360720']
nosy_count = 3.0
nosy_names = ['rhettinger', 'eric.snow', 'Hugo Ricateau']
pr_nums = []
priority = 'normal'
resolution = 'not a bug'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue39443'
versions = ['Python 3.8']

[HugoRicateau]

Assume one has defined the following descriptor:

class Descriptor:
    def __set__(self, instance, value):
        print('SET')

On the one hand, for the class-instance pair, the behaviour is as follows:

class FirstClass:
    descriptor = Descriptor()

    def __init__(self):
        self.descriptor = None

FirstClass().descriptor = None

results in "SET" being displayed twice; i.e. both assignations triggered the __set__ method of the descriptor.

On the other hand, for the metaclass-class pair, the behaviour is the following:

class SecondClassMeta(type):
    descriptor = Descriptor()

class SecondClass(metaclass=SecondClassMeta):
    descriptor = None

SecondClass.descriptor = None

results in "SET" being displayed only once: the first assignation (the one in the class definition) did not triggered __set__.

It looks to me like an undesirable asymmetry between the descriptors behaviour when in classes vs when in metaclasses. Is that intended? If it is, I think it should be highlighted in the descriptors documentation.

Best

[ericsnowcurrently]

First of all, thanks for asking about this. Everything is working as expected. Let's look at why.

First, be sure the behavior of descriptors is clear: the descriptor protocol is only triggered by "dotted access" on an object ("obj.attr"). So you should expect it only where you see that syntax used.

Let's look at your examples now.

FirstClass().descriptor = None

In this case there are 2 dotted accesses. The first one happens in __init__() when the object is created. The second is the rest of the above line.

class SecondClass(metaclass=SecondClassMeta):
descriptor = None

SecondClass.descriptor = None

In this case there is only one dotted access, in that last line. The object in this case is SecondClass and its class is SecondClassMeta. Unlike with FirstClass, the *class* in the second example (SecondClassMeta) does not have a __init__() with the dotted access. Instead there is only the one dotted access afterward. If SecondClassMeta had the same __init__() that FirstClass had then you would have seen a second trigger of the descriptor.

It seems you expected assignments (name binding) in the class definition body to be treated the same as dotted access. They are not. This is because when a class definition body is evaluated, the class object does not exist yet. The steps for class creation go like this:

1. figure out the metaclass (by default "type")
2. calls its __prepare__() method to get a namespace
3. execute the class body (like a function) with that namespace as the locals
4. create the class object, passing in that namespace

Python has worked this way since version 2.2 (PEP-252). See: https://docs.python.org/3/reference/datamodel.html#creating-the-class-object

If you want to get clever you could return a namespace object from your metaclass __prepare__ that triggers the descriptor protocol as you expected. However, I would not recommend that. Getting clever with metaclasses is best avoided. The default behavior is much simpler. That won't be changing.

It looks to me like an undesirable asymmetry between the descriptors behaviour when in classes vs when in metaclasses. Is that intended? If it is, I think it should be highlighted in the descriptors documentation.

Regardless, metaclasses are used infrequently and combining them with descriptors (especially relative to class definitions) is even less common. So pointing out the caveats of this case may not be worth the time of all future readers of those docs.

That said, clearly it would have helped you in this case. :) So here are some *possible* places to clarify (very briefly):

• descriptors howto
  + about mixing descriptors with metaclasses
  + a list enumerating places where descriptors are *not* invoked
• language reference (metaclasses section)
  + a warning saying something like "Avoid metaclasses if you can help it and only use them if you have a clear understanding of Python's object model (and dotted access)"
• language reference (descriptors/dotted access section)
  + a list enumerating places where descriptors are *not* invoked

Which of those do you think would have helped you the most?

[ericsnowcurrently]

@Raymond, What do you think about adding a helpful note or two in the docs?

[HugoRicateau]

Thanks for this detailed answer; very instructive :)

the descriptor protocol is only triggered by "dotted access"

Indeed; this is what I was missing... despite it is indirectly mentioned in the documentation. Nonetheless, it could be worth the overload to explicitly add in the language reference that 'the descriptor protocol is only triggered by "dotted access"' (looks like it is not the case for now).

• a list enumerating places where descriptors are *not* invoked
  [...]
  Which of those do you think would have helped you the most?

Could be really helpful as well, by clearly exhibiting the limitations of the descriptors; I think the best location for this could be the 'descriptors howto' page despite the other option is perfectly suitable as well.

Best,
Hugo

[rhettinger]

Some thoughts:

• The docs talk about descriptor invocation from "attribute access". The reason they don't say "dotted access" is that the descriptors can be invoked in multiple ways: dotted access, getattr()/setattr() functions, super(), or direct calls to __getattribute__().

• From the point-of-view of descriptors, metaclasses aren't special. The only essential fact needed is that type.__getattribute__() is called instead of object.__getattribute__(). The how-to guide already discusses how those methods differ.

• For now, I'm inclined to leave the docs as-is. The existing coverage of common cases is already a bit hard to read. It could become less readable if we list places where something doesn't happen, warnings to avoid features, or detailed explanations of uncommon cases like mixing metaclasses with descriptors. If this arises again, we could add a FAQ entry of some such.