help() on enum34 enumeration class creates only a dummy documentation

[andy-maier]

BPO	21561
Nosy	@terryjreedy, @ethanfurman, @serhiy-storchaka, @andy-maier
Files	bug1.py: Source file that produces the bug, using enum34 pydoc_fix2.py: fixed version of pydoc.py for Python 2.7.6; still has some debug prints; both fixes and debug code are marked with @am@ bug2.py: Simple standalone source file that produces the bug pydoc.py.patch: Patch for Lib/pydoc.py, relative to the tip of 2.7. test_pydoc.py.patch: Patch for Lib/test/test_pydoc.py, relative to the tip of 2.7.

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/ethanfurman'
closed_at = <Date 2020-05-31.14:43:42.789>
created_at = <Date 2014-05-23.17:27:11.248>
labels = ['type-bug', 'library']
title = 'help() on enum34 enumeration class creates only a dummy documentation'
updated_at = <Date 2020-05-31.14:43:42.788>
user = 'https://github.com/andy-maier'

bugs.python.org fields:

activity = <Date 2020-05-31.14:43:42.788>
actor = 'serhiy.storchaka'
assignee = 'ethan.furman'
closed = True
closed_date = <Date 2020-05-31.14:43:42.789>
closer = 'serhiy.storchaka'
components = ['Library (Lib)']
creation = <Date 2014-05-23.17:27:11.248>
creator = 'andymaier'
dependencies = []
files = ['35327', '35501', '35502', '35673', '35674']
hgrepos = []
issue_num = 21561
keywords = ['patch']
message_count = 13.0
messages = ['218979', '218982', '218990', '218993', '218994', '219088', '219404', '219599', '219894', '219895', '220861', '220862', '370469']
nosy_count = 4.0
nosy_names = ['terry.reedy', 'ethan.furman', 'serhiy.storchaka', 'andymaier']
pr_nums = []
priority = 'normal'
resolution = 'out of date'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue21561'
versions = ['Python 2.7']

[andy-maier]

Using the enum34 backport of enums, the help() function on an enum class Colors displays only:

-------
Help on class Colors in module __main__:

Colors = <enum 'Colors'>
-------

Source code to reproduce:
-------

from enum import Enum # enum34 package

class Colors(Enum):
    """docstring for enumeration Colors"""
    RED = 1
    GREEN = "2"
    BLUE = [3]

help(Colors)

---

Versions used:
python 2.7.6
enum34 1.0
Platform: Windows 7

I debugged the issue, found the place where it breaks down, and made a fix. However, it may well be that the fix is just a cure to a symptom, and that a better fix is possible.

Here is the fix:
In pydoc.py, class TextDoc, method docclass():
Change this code on line 1220+:
-------

            if thisclass is __builtin__.object:
                attrs = inherited
                continue
            elif thisclass is object:
                tag = "defined here"
            else:
                tag = "inherited from %s" % classname(thisclass,
                                                      object.__module__)

---

to this, by adding the two lines marked with "added":
-------

            if thisclass is __builtin__.object:
                attrs = inherited
                continue
            elif thisclass is object:
                tag = "defined here"
            elif thisclass is None:            # <-- added
                tag = "inherited from TBD"     # <-- added
            else:
                tag = "inherited from %s" % classname(thisclass,
                                                      object.__module__)

---

It breaks down during the last round through the 'while attrs' loop, where thisclass is None. I did not investigate why thisclass is None.

Without the fix, this causes an AttributeError to be raised by the classname() function, which then further on causes the dummy documentation to be generated.

With the fix, the help(Colors) output becomes:

-------
Help on class Colors in module __main__:

class Colors(enum.Enum)
 |  docstring for enumeration Colors
 |
 |  Method resolution order:
 |      Colors
 |      enum.Enum
 |      __builtin__.object
 |
 |  Data and other attributes defined here:
 |
 |  BLUE = <Colors.BLUE: [3]>
 |
 |  GREEN = <Colors.GREEN: '2'>
 |
 |  RED = <Colors.RED: 1>
 |
 |  

---

| Data and other attributes inherited from TBD:
|
| __members__ = {'BLUE': <Colors.BLUE: [3]>, 'GREEN': <Colors.GREEN: '2'...
-------

[ethanfurman]

Good work.

This bug was fixed in 3.4 with the inclusion of enum.

It would definitely be good to fix in 2.7 as well.

[ned-deily]

If the problem reported here applies only to the 2.7 backport of enum, which is not part of the Python standard library, shouldn't this issue be closed?

[ethanfurman]

The problem will affect anything that uses the same mechanism as enum. It also affects (not verified) all versions of python up to 3.4 where it was fixed because enum exposed it.

Besides which, I did not think a bug had to affect stdlib code in order to be fixed -- or this just a 2.7 restriction since it's basically end-of-lifed?

[ned-deily]

Sorry, I skimmed over the issue and didn't notice that the fix applied to pydoc.py, not enum.

[andy-maier]

The pydoc.py of Python 3.4 that supposedly has been fixed has a lot of changes compared to 2.7, but the place where I applied my "fix" in TextDoc.docclass() is unchanged.

So it seems that my fix should be regarded only to be a quick fix, and the real fix would be somewhere in the 3.4 pydoc.py. I tried to understand the changes but gave up after a while. My quick fix (with a better text than one that contains "TBD") is still better than not having it fixed, but more ideally the real fix should be rolled back to the 2.7 pydoc.py.

Is there anything else I can do to help with this bug?

[terryjreedy]

Devise a simple test (fail before, work after) that does not require enum34. If this fix is committed, the message could note that the same issue was fixed differently in 3.4 mixed in with other changes.

[ethanfurman]

I think something like the following, taken from http://bugs.python.org/issue19030#msg199920, shoud do the trick for something to test against:

    class Meta(type):
        def __getattr__(self, name):
            if name == 'ham':
                return 'spam'
            return super().__getattr__(name)

    class VA(metaclass=Meta):
        @types.DynamicClassAttribute
        def ham(self):
            return 'eggs'

which should result in:

    VA_instance = VA()
    VA_instance.ham  # should be 'eggs'
    VA.ham           # should be 'spam'

Combining all that with the DynamicClassAttribute should make a fitting test. Thanks, Andreas, for working on that.

[andy-maier]

Using Ethan's sample code (Thanks!!), I was pointed in the right direction and was able to produce a simple piece of code that reproduces the behavior without depending on enum34, as well as a proposal for a fix in pydoc.py.

The problem can be reproduced with a class that specifies a metaclass that has class attributes and includes these class attributes in its __dir__() result, which causes them to "propagate" as class attributes to the class using that metaclass, at least for the purposes of pydoc.

In the course of the processing within pydoc.py, the tuple returned by inspect.classify_class_attrs() for such class attributes then has None for its class item, which triggers the issue I originally reported (pydoc traps and produces just a dummy help description).

In my original problem with the enum34 module, the same thing happens, just here the "propagated" class attributes include the __members__ list, which is defined as a property. So I do believe that my simple code represents the same error situation as the original enum34 issue.

Because pydoc.py already has its own classify_class_attrs() function that wrappers inspect.classify_class_attrs() for the purpose of treating data descriptors correctly, I think it would be acceptable to continue down the path of fixing it up, just this case for class attributes propagated by the metaclass. That's what my proposed fix does.

Unfortunately, it seems one can attach only one file, so I paste the reproduction code in here, and attach the fixed pydoc.py.

Here is the code that reproduces the issue:
---------------- bug2.py
# Boolean test switches that control whether a class variable of the metaclass
# is added to the dir() result.
# If the fix is not present, then enabling each one triggers the error
# behavior; If none of them is enabled, the error behavior is not triggered.
with_food = True # Add the 'normal' class attribute 'food'
with_drink = True # Add the property-based class attribute 'drink'

class FoodMeta(type):
    """Metaclass that adds its class attributes to dir() of classes using it."""

    food = 'ham'      # 'normal' class attribute
    
    @property
    def drink(cls):   # property-based class attribute
        return 'beer'

    def __dir__(cls):
        ret = [name for name in cls.__dict__] # the normal list
        if with_food:
            ret += ['food']
        if with_drink:
            ret += ['drink']
        print "bug2.FoodMeta.__dir__(): return=%s" % (repr(ret),)
        return ret

class Food(object):
    """docstring for Food class"""
    __metaclass__ = FoodMeta

    def diet(self):
        return "no!"

if __name__ == '__main__':
    print "bug2: Calling help(Food):"
    help(Food)

---

-> Please review the reproduction code and the fix and let me know how to proceed.

Andy

[andy-maier]

Here is the bug2.py file pasted into the previous message, for convenience.

[andy-maier]

Attaching the patch for pydoc.py, relative to the tip of 2.7. the patch contains just the proposed fix, and no longer the debug prints.

[andy-maier]

Attaching the patch for Lib/test/test_pydoc.py, relative to the tip of 2.7. The patch adds a testcase test_class_with_metaclass(), which defines a class that provokes the buggy behavior, and verifies the fix.