creating stub pyi files automatically from pybind projects

[daltairwalter]

It would be nice if there was a way to automatically create stub .pyi files for intellisense from pybind11.

I am not sure if this is better done in pybind11 or in an external scraper utility that tries to analyze the dict of the pyd. It seems like there are some advantages if this could be done inside of pybind11 though.

1. Annotations and comments could be added in the primary source location.
2. It may be possible to provide more type information.
3. It could be used on embedded code instead of strictly being possible for pyd's.

[YannickJadoul]

My CLion/PyCharm IDE does a decent job though. Not sure how, but it does. Maybe look into that?

1. Annotations and comments could be added in the primary source location.

They already can be. pybind11 supports adding custom docstrings, etc.

2. It may be possible to provide more type information.

These are included as well as possible in the generated docstrings. Do you have any problems? Is there anything you can't do?

3. It could be used on embedded code instead of strictly being possible for pyd's.

I don't see how this would work. Embedded code, by definition, cannot be imported into Python. So why would you need .pyi files?

[daltairwalter]

I am really impressed by the docstring I see when I am querying it now in python. I haven't gotten tooling working to build pyi files yet, but this is really good. I am not sure how this will work if I also have custom docstring content, I really want to have both - though I haven't tested this yet.

I am not sure if this is officially supported by pybind11, but in general for embedded python PyImport_AppendInittab can be used to register modules that can then be imported from python code. It would be desirable to have pyi files for these modules.

[virtuald]

pybind11-stubgen works pretty well for generating pyi files, at least for the simple things I've tried personally. Apparently there are more complex cases where pybind11 gets names of things incorrect and then it breaks down -- but finding those bugs and fixing them would be great for everyone.

I'm currently working on autogenerating pyi for my project, the code I used (and some other thoughts) are there

[YannickJadoul]

I am not sure how this will work if I also have custom docstring content, I really want to have both - though I haven't tested this yet.

There are some docs on this: https://pybind11.readthedocs.io/en/stable/advanced/misc.html#generating-documentation-using-sphinx

I am not sure if this is officially supported by pybind11, but in general for embedded python PyImport_AppendInittab can be used to register modules that can then be imported from python code. It would be desirable to have pyi files for these modules.

Of course, it's possible. Again, it's in the docs. But you're not going to run mypy or anything over embedded modules (are you?), so I'm doubting the use of pyi files.
If not, you could export the embedded module as extension module, and generate pyi files, or something like that?

I'll close this issue, since it seems to be resolved? Feel free to continue the discussion, or maybe ask around on Gitter.

[daltairwalter]

This pybind11-stubgen project is really good. The pyi files it generates are working, though they have a lot of errors.

I think the errors are mostly coming from the docstrings though. Some C++ enum types are showing up in docstrings instead of the py::enum_ counterparts. I am using 2.4.3 though so maybe this is a known bug or has been fixed?

print(test.__dict__['Setup'].__dict__['getResourcePath'].__doc__)
getResourcePath(self: alt.hst.core.hstfitext.Setup, arg0: hstalgobase::RESOURCE_TYPE) -> str

[YannickJadoul]

Make sure you first expose a type (i.e., call the py::enum_ constructor), before using it as an argument in a function/method. If you don't, pybind11 doesn't know how to call the C++ type in Python.

[daltairwalter]

Thank you - exposing the type before using it as an argument was the problem. I am a bit surprised that this breaks the docstring but so far hasn't broken anything noticeably in execution.

[virtuald]

But you're not going to run mypy or anything over embedded modules (are you?), so I'm doubting the use of pyi files.

I have a project with embedded modules, and I've found that having a .pyi is more critical for embedded development because that's the only way I get autocomplete working. :-D

[YannickJadoul]

Thank you - exposing the type before using it as an argument was the problem. I am a bit surprised that this breaks the docstring but so far hasn't broken anything noticeably in execution.

Again, pybind11 doesn't know you're associating this C++ type with that Python name yet, so ... hard to generate the correct docstring, then.

I have a project with embedded modules, and I've found that having a .pyi is more critical for embedded development because that's the only way I get autocomplete working. :-D

Aha, right. Well, this embedded module will also contain the same docstring, I believe. It's just a lot harder because it's really a part of an executable, I suppose? How do you get out the docstrings/stub files, then?

[virtuald]

Well, this embedded module will also contain the same docstring, I believe. It's just a lot harder because it's really a part of an executable, I suppose? How do you get out the docstrings/stub files, then?

Yes, it's much harder to convince the autocomplete engine to execute the executable to extract the docstrings, which is why I do it separately. I have a part of my build process run the executable in a dummy mode that runs stubgen so it can extract the pyi out.

[sizmailov]

I think it's worth mentioning stubgen from mypy which can inspect binary python modules (not only pybind11-based).

At the moment mypy.stubgen produces better stubs in terms of correctness, while pybind11-stubgen is more informative but sometimes generates invalid output (usually caused by bad input). While for some cases pybind11-stubgen is good enough, I think it should be superseded (sizmailov/pybind11-stubgen#31) by mypy.stubgen in longer term.

[Priyatham10]

I have a project with embedded modules, and I've found that having a .pyi is more critical for embedded development because that's the only way I get autocomplete working. :-D

@virtuald Can you please lay out the process of automatically generating the .pyi stub files for embedded extension modules that have .pyd or .so extensions? Or, please point me to a good resource of your knowledge to do that. You can even provide the link of the project you used pyi files for your embedded modules. I have a python package in which we're shipping the extension modules with .pyd, .so. So, I want to give my package users the type hints and auto-completion when they import the package and consume symbols from the extension module in IDEs.

Or do you suggest writing them manually? There are enormous symbols which include classes, functions, parameters, etc.. from the binary extension module. So, I thought to generate the .pyi stubs automatically at first and then we can perfect it wherever necessary by examining it.

[daltairwalter]

Priyatham10, I am a little confused by the word embedded in your post. If you have a normal pybind11 .pyd or .so, then pybind11-stubgen works very well. mypy stubgen has been reported to work too. These tools are pretty straightforward to use and have there own documentation.

The embedded development that is being referenced is referring to a C++ application that has python modules embedded in the main executable - not in .pyd files. I think the stubgen programs are using the python module dictionaries to generate stubs so it may be possible to call one of these stubgen programs from within the C++ application, but I have not tried this.

[Priyatham10]

@daltairwalter Here embedded means i meant the binary extension module being provided as a python package.

[virtuald]

My project for autogenerating pybind11 wrappers and building a package is based on setuptools/distutils, and it's called robotpy-build (obviously I recommend it.. it makes creating pybind11 projects very simple!). After the build finishes (but before install runs), I have a distutils extension that generates pyi files via pybind11-stubgen. You may be able to use it standalone if you don't want to use robotpy-build.

https://github.com/robotpy/robotpy-build/blob/main/robotpy_build/command/build_pyi.py

[Priyatham10]

Hi @virtuald . I tried running the build_pyi.py file. But I'm confused as to what exactly should I provide for JSON of sys.stdin. I gave my package name as the first thing say: {"mapping": "delphifmx"} but I'm confused about whether I'm right or not. And also, should I give the output directory for the "out" key value? And what value should i give for the "stubs" key? Please tell me the format of what to be given for the input prompt..

Thank you so much..