Pybind generated pyd import does not resolve

[erinxocon]

Environment data

• Language Server version: 2020.6.1
• OS and version: Microsoft Windows Version 2004 (OS Build 20161.1000) Pre-release
• Python version (& distribution if applicable, e.g. Anaconda): Python 3.8.3

Expected behaviour

typing import pybind_generated_module would parse the module and generate intelisense responses when accessed.

Actual behaviour

Pylance informs me that Import "pybind_generated_module" could not be resolved Pylance (reportMissingImports)

Logs

IntelliCode: current invocation did not produce any meaningful tokens.

Other Information

I have the pybind module in my sitepackages folder. When using jedi or the microsoft langauge server, I am able to introspect my module and get intelisense completions. Not so when using Pylance. Pylance is fine completing other python based packages in the site package folder, and in extra folders I pass into the extra paths property. I can't include my module due to work things, but seeing as it works in the other two language server implementations, I'm assuming that this is related to pylance and not my module but you never know! My pyd/dll file has the proper classes in it that importing should be OK, I am able to use the module still in the script. Let me know what else I can do.

[FG] binding: c:\Users\erino\repos\pybind11\python\app.py (1ms)
[FG] parsing: C:\Users\erino\.virtualenvs\pybind11-dHr90HSR\lib\site-packages\ipython_genutils\__init__.py (3ms)
[FG] binding: C:\Users\erino\.virtualenvs\pybind11-dHr90HSR\lib\site-packages\ipython_genutils\__init__.py (1ms)
...

[erictraut]

Pylance/Pyright are not able to extract type information from binary files. This is as designed.

The recommended approach is to create or generate a type stub file that reflects the interface to the binary file.

[erinxocon]

This is great news! I can create the stub file. I just wasn't sure if it was a regression or intended since it worked in the Microsoft Language Server (is there a better name for that now?) Thanks for the swift response!

[erinxocon]

@erictraut when trying to go and define type stubs tonight I realized that pyright/pylance cannot even resolve the import. Even though the binary module is in the site packages folder, pylance does not pick up on it resulting in

Import "pybind_generated_module" could not be resolved Pylance (reportMissingImports)

I am fine defining stubs but I'm not sure how. I saw @savannahostrowski on discord today say that by setting type checking to strict, there would be a code action to create the stub, but I cannot get that to appear since my module is reported missing! Not parsing types out of the binary makes sense, however I would expect that it could find the module. The pyd file generated from pybind should have the correct PyInit_foo() method so I would expect that “import foo” would cause Pylance to search for foo.pyd, foo.py, and foo.pyc in the PYTHONPATH!

[erinxocon]

I'll amend my previous comment by saying that the modules are symlinked into the site packages folder. I tried adding the actual folder they were in to the extraPaths but it still wouldn't resolve the import. Let me know how else I can help.

[erinxocon]

I did a little more research on this issue tonight. I adapted my build script to package up the pyd file into a wheel and when I install that wheel pylance can indeed resolve the module. Still no auto complete cause it's all binary, but after defining some basic type stubs pylance does pick up on it! I'd still be curious to at least resolve binary files that can be imported into python (pyd or a dll, or whatever you used as long as it as the proper classes defined for python to call into), but given that the solution here is to turn the binary into a wheel, and pybind has examples on how to do that there is a workaround for now.

[JMVOLLE]

Pylance/Pyright are not able to extract type information from binary files. This is as designed.

The recommended approach is to create or generate a type stub file that reflects the interface to the binary file.

Hello. This really is a pity because jedi supports it (visual studio code) whithout the need for creating stub file. There is no straightforward way to create stubs using pybind11 (or I did not find one, any pointer greatly appreciated).

[jakebailey]

This is a tough problem because to do this, the module has to be executed to get info out of it, which may not be safe. We still want to find ways to support this, be it via explicitly asking for permission to do it, improving compiled module builders to make stubs, writing stubs, improving the maintenance of stubs next to compiled code, etc. It's just not simple, I'm afraid.

[judej]

Will need to wait for pybind to create stubs. Nothing for Pylance to do for now.