Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enhance inspect.getdoc to follow inheritance chains #59787

Closed
ncoghlan opened this issue Aug 8, 2012 · 26 comments
Closed

Enhance inspect.getdoc to follow inheritance chains #59787

ncoghlan opened this issue Aug 8, 2012 · 26 comments
Assignees
Labels
stdlib Python modules in the Lib dir type-feature A feature request or enhancement

Comments

@ncoghlan
Copy link
Contributor

ncoghlan commented Aug 8, 2012

BPO 15582
Nosy @ncoghlan, @merwok, @PCManticore, @ericsnowcurrently, @berkerpeksag, @vadmium, @serhiy-storchaka, @1st1
Files
  • inspect_getdoc.patch
  • issue15582.patch: Add documentation.
  • issue15582_1.patch: Ups, remove unintented change.
  • inspect_getdoc_inherited.patch
  • getdoc-news.patch
  • getdoc-news.v2.patch
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = 'https://github.com/serhiy-storchaka'
    closed_at = <Date 2015-07-30.15:12:21.015>
    created_at = <Date 2012-08-08.02:52:13.350>
    labels = ['type-feature', 'library']
    title = 'Enhance inspect.getdoc to follow inheritance chains'
    updated_at = <Date 2015-07-30.15:12:20.988>
    user = 'https://github.com/ncoghlan'

    bugs.python.org fields:

    activity = <Date 2015-07-30.15:12:20.988>
    actor = 'berker.peksag'
    assignee = 'serhiy.storchaka'
    closed = True
    closed_date = <Date 2015-07-30.15:12:21.015>
    closer = 'berker.peksag'
    components = ['Library (Lib)']
    creation = <Date 2012-08-08.02:52:13.350>
    creator = 'ncoghlan'
    dependencies = []
    files = ['33118', '34445', '34733', '38414', '39379', '40062']
    hgrepos = []
    issue_num = 15582
    keywords = ['patch']
    message_count = 26.0
    messages = ['167654', '167909', '206045', '215593', '232322', '232323', '234171', '237691', '237694', '237716', '237760', '240027', '240041', '240057', '240058', '240349', '241489', '241520', '241608', '241726', '241753', '242005', '243248', '247645', '247680', '247681']
    nosy_count = 9.0
    nosy_names = ['ncoghlan', 'eric.araujo', 'Claudiu.Popa', 'python-dev', 'eric.snow', 'berker.peksag', 'martin.panter', 'serhiy.storchaka', 'yselivanov']
    pr_nums = []
    priority = 'normal'
    resolution = 'fixed'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = 'enhancement'
    url = 'https://bugs.python.org/issue15582'
    versions = ['Python 3.5']

    @ncoghlan
    Copy link
    Contributor Author

    ncoghlan commented Aug 8, 2012

    Currently, if you override a method from a base class, the docstring is not inherited, even if it remains accurate.

    This issue proposes an enhancement to inspect.getdoc() that allows the docstring to be retrieved from the inheritance hierarchy in the case where it is not overridden in the subclass by providing an explicit docstring.

    Specifically, in the case where obj.__doc__ is None, and either the first parameter is a bound method, or a class object is passed in as the second parameter, inspect.getdoc will search the MRO based on obj.__name__ until it finds an attribute with a non-None __doc__ value.

    (In Python 2, this could have been automatic for both bound and unbound methods. Unfortunately, there are no unbound methods in Python 3, so the second parameter is needed to handle the unbound method case)

    @ncoghlan ncoghlan added stdlib Python modules in the Lib dir type-feature A feature request or enhancement labels Aug 8, 2012
    @ericsnowcurrently
    Copy link
    Member

    ericsnowcurrently commented Aug 10, 2012

    I have just the thing for this, but haven't had a chance to put a patch together. I might be able to get to it in the next week if Dave "bitey" Beazley isn't too much of a distraction. :)

    @PCManticore
    Copy link
    Mannequin

    PCManticore mannequin commented Dec 13, 2013

    Hello!

    Here's a patch. Currently it lacks doc updates, but if the solution is okay, then I could provide them.

    @PCManticore
    Copy link
    Mannequin

    PCManticore mannequin commented Apr 5, 2014

    Yury, Nick, how is my latest patch?

    @1st1
    Copy link
    Member

    1st1 commented Dec 8, 2014

    @Claudiu, you should also add this test and make sure that it passes:

       class Parent:
          __doc__ = 'some documentation'
    
       class Child(Parent):
          __doc__ = None

    assert getdoc(Child) is None

    In other words -- we use __doc__ defined in parent classes only when there was no __doc__ in children's __dict__s.

    @PCManticore
    Copy link
    Mannequin

    PCManticore mannequin commented Dec 8, 2014

    Alright, I'll update my patch later this week.

    @PCManticore
    Copy link
    Mannequin

    PCManticore mannequin commented Jan 17, 2015

    Yury, regarding your last message, is it actually possible to have a subclass which doesn't have a __doc__ attribute in its __dict__, except using slots? __doc__ seems to be set to None every time if it's not specified, so I don't know how could I detect the case where the client sets '__doc__ = None' himself.

    The following example might be more explanatory:

    >>> class A:
    ...    __doc__ = "a"
    ...
    >>> inspect.getdoc(A)
    'a'
    >>> inspect.getdoc(A())
    'a'
    >>> class B(A):
    ...   __doc__ = None
    ...
    >>> vars(B)
    mappingproxy({'__doc__': None, '__module__': '__main__'})
    >>> B.__dict__
    mappingproxy({'__doc__': None, '__module__': '__main__'})
    >>> class C(A): pass
    ...
    >>> vars(C)
    mappingproxy({'__doc__': None, '__module__': '__main__'})
    >>>

    Nevertheless, my patch ignores this case, since it operates only on methods. When trying to do inspect.getdoc(Child, parent=Parent), it will try to look for an attribute 'Child' in the mro of Parent and thus it will return None, since this doesn't exist (this can actually be a problem, if that attribute actually exist).

    @serhiy-storchaka
    Copy link
    Member

    serhiy-storchaka commented Mar 9, 2015

    The patch adds support only for unbound methods and requires additional parameter for this. It is not what should be done in this issue at all.

    I'm interested in this issue so I'll write a patch. It should be easier to write it myself than describe it.

    @serhiy-storchaka serhiy-storchaka self-assigned this Mar 9, 2015
    @PCManticore
    Copy link
    Mannequin

    PCManticore mannequin commented Mar 9, 2015

    The patch adds support only for unbound methods and requires additional parameter for this.

    Just curios, which part works only for unbound methods?

    It is not what should be done in this issue at all.
    It should be easier to write it myself than describe it.

    Well, some hints for the right way to do this would have been appreciated.

    @serhiy-storchaka
    Copy link
    Member

    serhiy-storchaka commented Mar 9, 2015

    Here is a patch that supports classes, bound and unbound methods, class methods, properties, method and class descriptors.

    @ncoghlan
    Copy link
    Contributor Author

    ncoghlan commented Mar 10, 2015

    Serhiy's patch looks good to me.

    I'd completely missed that __qualname__ could be used to avoid needing a second argument even when handling objects other than bound methods. That's very cool, and I can see why you figured it was easier to just write the patch than explain what you had in mind.

    @python-dev
    Copy link
    Mannequin

    python-dev mannequin commented Apr 3, 2015

    New changeset 157b7055bb9d by Serhiy Storchaka in branch 'default':
    Issue bpo-15582: inspect.getdoc() now follows inheritance chains.
    https://hg.python.org/cpython/rev/157b7055bb9d

    @serhiy-storchaka
    Copy link
    Member

    serhiy-storchaka commented Apr 3, 2015

    Unfortunately this enhancement breaks test_enum.

    ======================================================================
    FAIL: test_pydoc (test.test_enum.TestStdLib)
    ----------------------------------------------------------------------

    Traceback (most recent call last):
      File "/home/serhiy/py/cpython/Lib/test/test_enum.py", line 1609, in test_pydoc
        self.assertEqual(result, expected_text)
    AssertionError: 'Help[68 chars]n |  Generic enumeration.\n |  \n |  Derive fr[876 chars]ing.' != 'Help[68 chars]n |  Method resolution order:\n |      Color\n[782 chars]ing.'
      Help on class Color in module test.test_enum:
      
      class Color(enum.Enum)
    -  |  Generic enumeration.
    -  |  
    -  |  Derive from this class to define new enumerations.
    -  |  
       |  Method resolution order:
       |      Color
       |      enum.Enum
       |      builtins.object
       |  
       |  Data and other attributes defined here:
       |  
       |  blue = <Color.blue: 3>
       |  
       |  green = <Color.green: 2>
       |  
       |  red = <Color.red: 1>
       |  
       |  

    | Data descriptors inherited from enum.Enum:
    |
    | name
    | The name of the Enum member.
    |
    | value
    | The value of the Enum member.
    |
    | ----------------------------------------------------------------------
    | Data descriptors inherited from enum.EnumMeta:
    |
    | __members__
    | Returns a mapping of member name->value.
    |
    | This mapping lists all enum members, including aliases. Note that this
    | is a read-only view of the internal mapping.

    ----------------------------------------------------------------------

    @python-dev
    Copy link
    Mannequin

    python-dev mannequin commented Apr 4, 2015

    New changeset 47a61a1c97b3 by Serhiy Storchaka in branch 'default':
    Fixed test_enum for issue bpo-15582.
    https://hg.python.org/cpython/rev/47a61a1c97b3

    @serhiy-storchaka
    Copy link
    Member

    serhiy-storchaka commented Apr 4, 2015

    Changed test_enum to make buildbots green, but perhaps the docstring of Enum should be changed, because it now is used for all Enum subclasses that doesn't define a docstring explicitly. An alternative solution is to set __doc__ of Enum subclasses to an empty string if the docstring is not defined explicitly.

    @ncoghlan
    Copy link
    Contributor Author

    ncoghlan commented Apr 9, 2015

    I filed issue bpo-23900 to consider the question of the default docstring for Enum subclasses.

    @vadmium
    Copy link
    Member

    vadmium commented Apr 19, 2015

    Sometimes the doc string for the overridden method does not make much sense in the context of the subclass. Just wondering if this was considered; it seems like a fairly serious downside to this new feature. E.g. in a package I am reviewing, there is a class that inherits HTMLParser and converts HTML to PDF. There is no doc string, so previously there was just the signature in the “pydoc” output. Now the output looks like:

    | __init__(self, pdf, image_map=None)
    | Initialize and reset this instance.
    |
    | If convert_charrefs is True (the default), all character references
    | are automatically converted to the corresponding Unicode characters.

    The second paragraph mentions parameters and settings of the internal base class, which doesn’t make much sense for the subclass.

    @ethanfurman
    Copy link
    Member

    ethanfurman commented Apr 19, 2015

    Sounds like good incentive to add good docstrings. :)

    @serhiy-storchaka
    Copy link
    Member

    serhiy-storchaka commented Apr 20, 2015

    I were aware that this can propagate some not well appropriate docstrings from abstract base classes, but Martin expose worse problem: inheriting a docstring by the method with changed signature.

    Perhaps we should check if a signature of overriding method is compatible with a signature of base method. But this is hard to implement, and impossible in some cases until all builtins will converted to Argument Clinic.

    @ethanfurman
    Copy link
    Member

    ethanfurman commented Apr 21, 2015

    I know it will take some time to add doc strings to places where they are missing, but I would not roll-back the patch for that. File new issues and get the over-riding methods properly documented.

    @vadmium
    Copy link
    Member

    vadmium commented Apr 21, 2015

    My example was taken from a package that tends to “properly document” everything in separate documentation files, rather than in the source code. I don’t really think Python should dictate if and how one document their code based on what base classes they use, and pydoc is still useful with code that has no doc strings.

    But if this is to be the way of the future, perhaps a warning in the “What’s New” page might be a good idea. The inspect.getdoc() documentation should probably also have a “Changed in version 3.5” warning.

    I guess it would be less automatic, but maybe another option might have been to add an @inherit_docstring decorator, or some explicit way to say a particular API (re)implements an (abstract) API documented elsewhere.

    @ncoghlan
    Copy link
    Contributor Author

    ncoghlan commented Apr 25, 2015

    Yes, this was a deliberate change to "flip the default" as to which subclasses get bad docstring behaviour.

    In the status quo, if you provide a subclass method which does basically the same thing as the parent class, you lose the docstring unless you duplicate it, meaning you have to choose between bad docstrings and a maintainability problem due to content duplication.

    With this change, you only need a custom docstring in the subclass if you've changed the method behaviour enough that the parent class docstring no longer makes any sense.

    If you just want to suppress the docstring entirely, then you'll need to specify an empty docstring.

    This is potentially worth a note in the "Porting to Python 3.5" section of the What's New document, but it's an intended consequence of the change.

    @vadmium
    Copy link
    Member

    vadmium commented May 15, 2015

    getdoc-news.patch suggests some wording to add to What’s New, and also adds a “Changed in version 3.5” note to inspect.getdoc().

    BTW I also noticed that the class doc strings are not inherited from object.__doc__, although method doc strings _are_ inherited from object(), such as object.__init__.__doc__. The current documentation suggests that the class doc string “The most base type” should also be inherited.

    $ cat module.py
    class SomeClass:
        '''CLASS DOCSTRING'''
        def __init__(self):
            '''METHOD DOCSTRING'''
    $ ./python -m pydoc module.SomeClass  # Doc strings intact
    [. . .]
    module.SomeClass = class SomeClass(builtins.object)
     |  CLASS DOCSTRING
     |  
     |  Methods defined here:
     |  
     |  __init__(self)
     |      METHOD DOCSTRING
     |  [. . .]
    $ ./python -OOm pydoc module.SomeClass  # Method inherited, class stripped
    [. . .]
    module.SomeClass = class SomeClass(builtins.object)
     |  Methods defined here:
     |  
     |  __init__(self)
     |      Initialize self.  See help(type(self)) for accurate signature.
     |  [. . .]

    I also wonder how well this feature would work when someone tries to override a base method by using a mix-in type class.

    @vadmium
    Copy link
    Member

    vadmium commented Jul 30, 2015

    Merged the current What’s New page in getdoc-news.v2.patch. Is there any interest in applying this?

    @python-dev
    Copy link
    Mannequin

    python-dev mannequin commented Jul 30, 2015

    New changeset 4476b578b8fd by Berker Peksag in branch '3.5':
    Issue bpo-15582: Add a whatsnew entry for inspect.getdoc() changes in 3.5.
    https://hg.python.org/cpython/rev/4476b578b8fd

    New changeset e0f4a5f09bfa by Berker Peksag in branch 'default':
    Issue bpo-15582: Add a whatsnew entry for inspect.getdoc() changes in 3.5.
    https://hg.python.org/cpython/rev/e0f4a5f09bfa

    @berkerpeksag
    Copy link
    Member

    berkerpeksag commented Jul 30, 2015

    Applied, thanks!

    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    stdlib Python modules in the Lib dir type-feature A feature request or enhancement
    Projects
    None yet
    Development

    No branches or pull requests

    7 participants