[Feature Request] add support for type annotation hints in #[text_signature] macro.

[robinmoussu]

Would it be possible to extend the text_signature macro to be able to support python type annotation.

Example

/// Formats the sum of two numbers as string.
#[pyfunction()]
#[text_signature = "(a: int, b: int) -> str"]  //  <-- this line
fn sum_as_string(a: usize, b: usize) -> PyResult<String> {
    Ok((a+b).to_string())
}

Expected documentation:

sum_as_string(a: int, b: int) -> str
        Formats the sum of two numbers as string.

• All arguments should be able to have a type annotation
• The return type of the function could be added with ->type
• According to the PEP any expression should be valid, but I think that limiting it only to types (and maybe quoted string) should be enough
• I think that the types should be python types (python's int instead of rust's i64 for example)
• Types used in the annotations don't need to be checked in the macro
  • In the example, I used int for the annotation of a and b even if Rust expect an usize since python doesn't have unsized integers
  • In the example, the return type of the function is just str and not PyResult

Note

After reading #310 (comment) I think that it's not possible to use the same machinery than type annotation in python code (which, from what I understand, would allow IDE to give better auto-completion hints). While waiting for the cpython bug to be fixed, it should still be possible that the text returned by help(sum_as_string) contains the type annotation even if __annotations__ isn't populated correctly (yet).

[davidhewitt]

I'm not sure that text_signature is the right place for annotations, because the C specification they're supposed to be compatible with doesn't support them. But I am definitely in favour of adding support for annotations to pyo3 (I even hope that they could be automatically generated one day)!

[LucaCappelletti94]

Is there any update on this issue?

[davidhewitt]

Afraid not; proposals for design and implementation are always welcomed.

[LucaCappelletti94]

Hello @davidhewitt, would it be possible to plan for a call to discuss what has already been explored and what is yet to be attempted?

[CaffeineDuck]

Type annotations in pyo3 would sure be very helpful. But is there some hack around it for now? I am creating a python module in rust and I really do need editor auto-completion support.

[sbdchd]

@CaffeineDuck What I've done in the past is unpack the .whl and copy over some manually created stub files (.pyi).

essentially:

./.venv/bin/wheel unpack foo-0.1.0.whl
touch foo/py.typed
cp -R ./foo foo-0.1.0
./.venv/bin/wheel pack foo-0.1.0

where there are some .pyi files in ./foo

[CaffeineDuck]

@sbdchd Dats big brain, thanks.

[davidhewitt]

Note that if you're using maturin or setuptools-rust, both of them can put your .pyi files into the wheel directly on first build rather than needing to patch it manually.

[davidhewitt]

@LucaCappelletti94 if you want you can reach out to me on gitter and we can try to arrange something. TBH the short answer is that I'd really love to have type annotations supported better in PyO3, however I haven't thought about them much myself and there's other things I'm working on at the moment.

Also I prefer to share information on GitHub issues rather than calls because then it's available for everyone to read it!

[LucaCappelletti94]

@davidhewitt yeah I absolutely concur that the information must be shared on GitHub issues, we could just keep a small record of the important points, if any, discussed during the proposed call. The call would be only a possibly shorter attempt to sync up and avoid trying stuff that you have already attempted. I'll reach to you on Gitter!

[LucaCappelletti94]

Logging here what was briefly discussed on Gitter that I believe could be of help to other users:

1. Currently, the best way to make type hinting available is to write .pyi files. An example of a .pyi file can be found here.
2. Files with that format are automatically incorporated in the wheel when using maturin.
3. We (me and colleagues) will focus on developing a tool to automatically generate such files as we already automatically generate most of the other python bindings. If the resulting pipeline is not too hacky and crumbly, we will try to make it available for other projects.

[CLOVIS-AI]

For multiple reasons, using .pyi files is inconvenient for me at the moment (help(foo) ignores them, it's in a different place from the code so no one updates them, etc).

Until it's possible for PyO3 to generate the type annotations automatically, would it be possible to have something similar to text_signature that would accept a string and place a # type: comment at the top of the function? Something like this:

/// Documentation here
#[pyo3(text_signature = "(a, /)", type = "(int) -> Optional[int]")]
pub fn foo(a: u64) -> Option<u64> {
    todo!()
}

which would generate:

def foo(a):
    # type: (int) -> Optional[int]
    """Documentation here"""
    pass

This syntax (https://peps.python.org/pep-0484/) is supported by MyPy and most IDEs. PyO3 doesn't even need to check that it's correct. It's compatible with all versions of Python.

Would this be doable? I can submit a PR if someone guides me through what needs to be changed.

[davidhewitt]

The problem isn't the PyO3 syntax but rather how to transfer this information from the Rust source through to Python. The only space for metadata that the Python C-API gives us to communicate information is the text_signature, but that's got strict rules on its syntax and doesn't support annotations.

I've been considering reaching out to upstream CPython to request the text_signature specification be expanded to allow annotations, but hadn't got around to it yet. I think that would be the cleanest solution. There are other possibilities such as building a separate tool to parse the Rust source and generate a .pyi file, however nobody has tried implementing AFAIK.

[CLOVIS-AI]

Ah, I see. Would it be possible to integrate such a tool with PyO3 so it can take advantage of the information PyO3 already possesses? (e.g. as an optional cargo feature).

I guess if it's integrated well enough with PyO3 and Maturin it can work just as well as if the information was included in the objects directly. The only downside I see is that from my experimentation, help(object) ignores the .pyi files (but MyPy and PyCharm find them without issues). Using stub files may even be better for things like HTML generation of documentation.

[davidhewitt]

I don't think it would integrate with cargo build but I can imagine maturin would be able to run such a tool when packaging, yes.

[CLOVIS-AI]

The .pyi file generation itself should be quite simple (all the information is already available to PyO3 if we add the #[type]).I can try to prototype this, but I'm not familiar with PyO3's codebase to know where to take the information from.

[davidhewitt]

In theory you might be able to use pyo3-macros-backend for the parsing. I'm actually working on a new signature implementation in #2302 and would be happy for it to parse annotations.

[CLOVIS-AI]

I attempted to use the macros to generate the info, but they don't have access to type information so it's a bit limited.

I'm currently looking at rustdoc's JSON output, which contains the list of all impl blocks, to see if I can find out how each type is converted to/from Python.

[davidhewitt]

👍 that might save us having to do the parsing too, good idea.

If you find you need to add extra sentences to the docs for FromPyObject implementations like "the type annotation for this is List[T]", that would probably be helpful for users anyway!

[CLOVIS-AI]

I'll update you on what I manage to do. I'll need you so Maturin can start that tool anyway :)

[CLOVIS-AI]

I forgot to link it here, and I see that many people follow this issue. I'm prototyping a way to generate python stubs (pyi), hopefully without needing any annotations. Please take a look and share your feedback here: #2379

[fzyzcjy]

Hi, is there any updates? Thanks

[davidhewitt]

In PyO3 0.18 we will autogenerate text_signature, however the request here requires upstream CPython support, someone needs to champion it upstream to make it possible.

[messense]

xref: python/cpython#101872

[davidhewitt]

Unfortunately annotations in text_signature was generally rejected at the PEP draft level in CPython, so I can't see us adding to our macro for the forseeable future. Will close this.

https://discuss.python.org/t/type-signatures-for-extension-modules-pep-draft/43914