[Feature Request] Support for extending documentation using ?

[nishikantparmariam]

IPython's feature to show documentation using foo.bar? is very helpful. It would be great feature addition to provide support for dynamically showing documentation when say bar in not an actually property (defined with @property and having docstring) or attribute on foo, but can be set on foo.

Solution for showing dynamic tab completions

As an example, for showing tab suggestions __dir__ can be defined on a class as per #7245 (comment). But, there does not seem an equivalent to make docstring using ? also work.

Possible solution

A custom hook can be provided (may be with better name) which users can use something like -

def show_documentation(self, obj, attrname): 
    """
    :return:
         Docstring of attribute `attrname` on object `obj`
    :rtype: 
         str
    """

So, if someone does foo[2].bar.baz? and if baz is not found with @property on bar, then show_documentation(self, <bar object (preferably) or class of bar object>, 'baz') is invoked.

Additionally, it would be great if after doing say foo[2].bar.baz = 5 it continues to show the above documentation and not the documentation for int.

This is just one solution, open for better ideas!

[Carreau]

Apologies for the late reply.

If my understanding is correct, you want o be able to define

class Bar:

    def show_documentation(self, obj, attrname): 
        """
        :return:
             Docstring of attribute `attrname` on object `obj`
        :rtype: 
             str
        """
        if obj is self and attrname is 'baz':
           return "this is a baz it is the bazified value of the Bar instance
        

Ir you you want the hook to be external:

ip = get_ipython()
ip.register_doc_hook(...)

I do have some concerns about the first instance, similar things have longed been discussed, and the consensus is that we do not want to have object-controlled code execution when showing documentation. This is a security risk, and if documentation can start to have side effects it will create other problems, like impossibility to see documentation when the kernel is busy. This is already a problem when even with just __doc__.

For the second This is a larger project (https://github.com/jupyter/papyri) I am working on that completely rework how current documentation work, and would allow more dynamic (and html browsable) documentation within notebook. Though this is not a small project.

I think you are more meaning the second one, but I'm slightly worried of having it be a cusotmizable hook. Can you give me an example of why you would like it to be different on a per object basis ?

Would something like simply looking up __<attrname>_doc__ be sufficient for you / why not ?

[Carreau]

One other issue I can have is that with the above,

foo.bar? and x = foo.bar; x? can now return two different documentation as in the second case we have no way of discovering wether x (.bar) has documentation attached to it, and this may be really confusing. So I'm not too sure how to handle this.

[nishikantparmariam]

Thanks for the response. What I wanted was some way to know -

• The object on which documentation was requested i.e. object just before the last dot, bar object in case of foo[2].bar.baz? - to not show documentation on arbitrary objects but only on objects of class Bar.
• Name of the attribute requested (baz)

Then, we will do "minor computation" and return a string.

---

Yes, initially I thought it could be an external hook that receives the object and the attribute name. But, given the points you mentioned above I think we can explore this simple solution -

Would something like simply looking up __attrname_doc__ be sufficient for you / why not ?

I am assuming you mean defining a class variable. This may not be sufficient because the no. of attributes can be high and we do not want to hard-code it on the class. Maybe a bit more generic than this -

class Bar:
    __custom_documentations__ = (Dict-like-object)

So, somewhere here, it can check if __custom_documentations__ is defined on Bar and if yes, then use __custom_documentations__[attributename] (We have to be cautious as attributename may not be in __custom_documentations__).

In this case, user (we) would have to do something like below, but it should be fine -

class DictLike:
    def __getitem__(self, k):
         ... "the minor computation at runtime to generate docstring" ..
         return "documentation for k"
class Bar:
    __custom_documentations__ = DictLike()

Above solution still limits it to class level, it would be much better if we have it at object level e.g. defining a custom method show_documentation(self, ..) on the object.

if documentation can start to have side effects it will create other problems

Totally agree here. We should leave it totally up to the user to not have side effects / execute arbitrary code if they are using this feature. getdoc method is one such example, where user can execute arbitrary code on foo?, but it should be their responsibility.

foo.bar? and x = foo.bar; x? can now return two different documentation as in the second

This should be fine. We need not handle the second case x?. For example, foo.bar? shows a custom documentation (both before and after the user does say foo.bar = 5), but when user does x = foo.bar; x?, it is fine that it shows default documentation of int in this case. It should behave as if bar was a property on class Foo with some docstring.

---

Opens for suggestions!! cc: @mlucool if he has some thoughts on this.

[Carreau]

Thanks for your patience. I've been looking at the code, and it looks like this should be doable with some refactor, I need to thread some extra information here and there and have some refactor, but it seem the code already has a notion of object and its parent. in the case of foo[2].bar.baz?, the "parent" being "foo[2].bar".

This information is just not handled everywhere, but once that's done, it shoudl be straitforward to look at wether parents has the right fields/methods and dispatch to it.

There is some minimal complexity due to sphinxify that tries to convert docs to html, and minor changes to public methods. I'll do some refactor/cleanup first.

[Carreau]

Misc questions/interogation while I'm working on it:

• Should the return docs still include the information of the property fields when they still exists, that is to say, if bar.foo is a DataFrame, is it still useful to show that docs ?
• How much should the hook be able to override, or do you simply want to display the docstring ? (currently the type, and a few other things are infered from the object itself.
• There are currently 3 things operation magics that this affects: pinfo/?, pinfo2/?? and pdoc, I'm assuming you only want to have it support ? so far, yes ?

[nishikantparmariam]

Thanks @Carreau for working on this.

Should the return docs still include the information of the property fields when they still exists, that is to say, if bar.foo is a DataFrame, is it still useful to show that docs?

It should always show docstring from __custom__documentations__ (if it exists), even if attribute is added later, similar to how it works with @property e.g.

class Bar:
    
    @property
    def foo(self):
        """
        Docstring for foo
        """ 
        return self._foo

    @foo.setter
    def foo(self, v):
        self._foo = v
        
bar = Bar()
bar.foo? # Gives 'Docstring for foo'

Now, doing below still gives 'Docstring for foo' and not for a DataFrame's docstring:

bar.foo = DataFrame(range(10))
bar.foo?

So, with __custom__documentations__ also it should behave similarly e.g.

class DictLike:
    
    def __getitem__(self, k):
        if k == 'foo':
            return "Docstring for foo"
        raise KeyError

class Bar:    
    __custom__documentations__ = DictLike()    
    
bar = Bar()
bar.foo? # Gives 'Docstring for foo'

Now below should still query __custom__documentations__ (because it exists on parent) instead of showing docstring of object foo (which is defined now):

bar.foo = DataFrame(range(10))
bar.foo? # Should still give 'Docstring for foo'

This way it will be consistent before and after doing bar.foo = ... Let me know if I misunderstood your question.

How much should the hook be able to override, or do you simply want to display the docstring ? (currently the type, and a few other things are infered from the object itself.

This is an interesting thing. For now, simply docstring should do and type can be property ? Because the object foo does not exist yet, e.g.

class DictLike:
    def __getitem__(self, k):
        if k == 'foo':
            return "Docstring for foo"
        raise KeyError

class Bar:    
    __custom__documentations__ = DictLike()    
    
bar = Bar()

Now doing bar.foo? should produce below kind of output both before and after bar.foo = DataFrame(range(10)) is done.

Type:        property
String form: <property object at 0x7efd0c01a4d0> (not sure about this line..?)
Docstring:   Docstring for foo

There are currently 3 things operation magics that this affects: pinfo/?, pinfo2/?? and pdoc, I'm assuming you only want to have it support ? so far, yes ?

Yes, only pinfo/? should be fine for now.

---

Open to other ideas and thoughts!!

[nishikantparmariam]

Closing as this was released with IPython 8.12. Thanks @Carreau.