Set the __text_signature__ attribute of callables

[anntzer]

Issue description

On recent enough Python versions (I think 3.4+), non-overloaded Python functions created by pybind11 could set the __text_signature__ attribute to a textual representation of the function signature, instead of prepending it to the docstring (see inspect._signature_from_builtin in the stdlib). This attribute is recognized (and parsed) by at least pydoc and inspect.signature (and thus, I guess, Sphinx too, though I am not sure), so it would perform the task of communicating the function signature better than the current method.

(Originally posted on gitter, it was suggested that I put the proposal here.)

Reproducible example code

N/A

[dean0x7d]

Thanks for pointing this out. I've been doing some research into function signatures in Python 3.x.

TL;DR Bad news: We can't add __text_signature__ because the format is incompatible with pybind11. Good news: We should be able to add __signature__ which would accomplish the same goal but also include even more information (type annotations).

There are a couple of issues with __text_signature__ which make it a bad fit for pybind11. The format does not support annotations which we use for type information. The default __text_signature__ is a property which parses the docstring for the hardcoded signature marker which assumes there is no return type annotation. In contrast, pybind11 generates signatures like: add(a: int, b: int) -> int. There's also the fact that __text_signature__ isn't officially documented which might make it a bit uncertain. Although, the lack of type annotations is already a deal breaker.

Instead, I think we can make use of __signature__ which is well documented in PEP 362 and available since Python 3.3. This was sort of the plan before, but hasn't quite materialized yet. I'll try to put it together now (__signature__ can be a property which lazily generates the Signature object by parsing our docstring format). Implementing this should also resolve issue #990.

[aldanor]

@dean0x7d I wonder if you ever had a chance to look at it?

I had a brief look, it would probably be the easiest to do it in pure Python rather than C++ (i.e. parse the signature strings that we generate in C++), but I'm not sure that would be acceptable.

[anntzer]

A side benefit of using __signature__ instead of tucking the signature into __doc__ is that as of Py3.5+, inspect.getdoc (used by pydoc, sphinx, etc.) implements docstring "inheritance" (https://docs.python.org/3/library/inspect.html#inspect.getdoc).

So say for example there is an abstract base class defined in Python:

class Base:
    def foo(self): "some docstring"

and an implementation of it using pybind11 that reimplements foo without setting a signature, then right now the docstring of foo (as rendered by pydoc or sphinx) would be the pybind11-generated signature, whereas it would be the docstring from the base class if pybind11 did not write its own __doc__.

[anntzer]

I had a quick stab at it and it is in fact relatively easy to construct the signature object in initialize_generic (ignoring for now what we do with overloaded functions...), at the same time as the docstring-signature is constructed.

The real problem is that there is nowhere to actually stash this information, because PyCFunctions do not have (AFAICT?) a __dict__ (as can be seen from the fact that one cannot, from Python, assign an attribute to a pybind11 function). I believe that Cython works around the issue by creating a custom callable type instead of PyCFunctions (https://github.com/cython/cython/blob/master/Cython/Utility/CythonFunction.c), but that's arguably a much more complicated project...

[anntzer]

Looks like this should wait until https://www.python.org/dev/peps/pep-0575/ (possibly Py 3.8) which will give us a __dict__ on C functions too. Closing as unfixable for now.

[martinwicke]

Since PEP575 was withdrawn, is it sensible to pursue the __text_signature__ path?

I agree that __signature__ would be much nicer, but I'll take the generated docstring over nothing at all. The generated string to stash in __doc__ would have to be modified (sadly, stripped of useful information), but that's still feasible?

[phcerdan]

For the regular user, in Linux, you can import rlcompleter and you will get TAB completion on members. After reading the docs of rlcompleter seems to only work out of the box in Unix-like OS's.

[riddell-stan]

(I'm porting code from Cython to pybind11.)

I need to know the parameter names for a function. (Previously, in Cython, inspect.Signature.from_callable provided these.) Is the current best solution to write some code to parse __doc__?

[YannickJadoul]

@riddell-stan, I believe/am afraid it is; the C functions pybind11 exposes to Python do not work with inspect, as far as I know. Any idea how Cython accomplishes this?

[riddell-stan]

@YannickJadoul Thanks! The relevant code wasn't too difficult to write.

The relevant Cython compiler directive is described here: https://cython.readthedocs.io/en/latest/src/userguide/source_files_and_compilation.html#compiler-directives

binding (True / False) Controls whether free functions behave more like Python’s CFunctions (e.g. len()) or, when set to True, more like Python’s functions. When enabled, functions will bind to an instance when looked up as a class attribute (hence the name) and will emulate the attributes of Python functions, including introspections like argument names and annotations.

[YannickJadoul]

@riddell-stan Right, thanks for that reference (never really worked with Cython before) :-)

Reading this, I think pybind11 always goes with the first of those two possibilities :-/

[pitrou]

For the record, the way __text_signature__ is looked up on CFunctions is to parse the first line of the docstring.
You can see an example here:
https://github.com/python/cpython/blob/master/Modules/clinic/_bisectmodule.c.h#L6

>>> bisect.bisect_right.__text_signature__
'($module, /, a, x, lo=0, hi=None)'
>>> inspect.signature(bisect.bisect_right)
<Signature (a, x, lo=0, hi=None)>

[YannickJadoul]

@pitrou, hmmm, interesting! Any idea if/how this can work together with the current situation, where we put the full signature (including return type) on the first line? For example, Sphinx autodoc picks up this first line, I believe.

[pitrou]

I think Sphinx autodoc picks up the generated __text_signature__ automatically (but you'll need to check that :-)). So will IPython and the built-in prompt's help() function.

[YannickJadoul]

But we do lose the signature's information about the return type? :-/

[pitrou]

Hmm, that is an interesting concern. I had never thought about that.

[YannickJadoul]

Apart from that, I'd love to look into it! If there's a way to provide this, and make all pybind11 functions more native-like, that would be amazing.

There were a few other concerns, though, such as the fact that we cannot include type annotations (if that's still correct?): #945 (comment)
Those are also quite important to pybind11 users, I would imagine, since it provides the interface to the C++ types and indications on why an invocation fails.

Ideally, it would be great to support both the current signature as well as __text_signature__.

[pitrou]

Here is how I gather things work, based on a real example cmath.isclose.

At the C level, cmath.isclose defines the following "docstring" (i.e. the PyFunctionObject's func_doc C member):

isclose($module, /, a, b, *, rel_tol=1e-09, abs_tol=0.0)
--

Determine whether two complex numbers are close in value.
[...]

But this is not what is exposed at the Python level, because __doc__ and __text_signatures__ go through descriptors which parse the physical func_doc and extract the text signature in compliant format (the -- on the second line may be used as separator):

>>> print(cmath.isclose.__text_signature__)
($module, /, a, b, *, rel_tol=1e-09, abs_tol=0.0)
>>> print(cmath.isclose.__doc__)
Determine whether two complex numbers are close in value.
[...]

So I think that you could add your own signature below the generated text-signature, and your own signature would remain at the top of the Python-visible docstring, for example:

isclose($module, /, a, b, *, rel_tol=1e-09, abs_tol=0.0)
--

isclose(a: complex, b: complex, *, rel_tol: float=1e-09, abs_tol: float=0.0) -> bool

Determine whether two complex numbers are close in value.
[...]

Do note that you cannot experiment on this with Python objects. The automatic extraction only happens on PyCFunctionObject.

[pitrou]

Ok, I asked, and you should be able to use type annotations in the text signature, including for the return type (worth checking it for real though :-)).

[YannickJadoul]

@pitrou, thank you so much! I'll look into this, after we get the last few bugs out for 2.6.0! :-)

[diiigle]

I am coming from davidhalter/jedi#1388 and digging around this topic in the last few days.

Stating the obvious, but I want to highlight how valuable this addition would be to all downstream projects! Autocompletion in python (only) works pretty well these days, but as soon as it hits binary modules (from pybind11) many language servers seem to struggle (Jedi, Pylance, microsoft-language-server).

Explicit signature information through the __signature__ attribute might solve this problem across the board. So here is a bump from me.

[fvannee]

+1 on this

[OmerShapira]

Strong +1 . Getting explicit type annotations will help tremendously with frameworks that use pybind classes to export functions from python to other computing contexts.

[Beilinson]

+1, strong autocompletion is standard in every ide and for every language and library, binary modules must support it as well.