cython compatibility

[munechika-koyo]

I work on python-based programming using a certain class implemented by cython.
Although the conventional language server (e.g. Microsoft python language server) can show the docstrings written in a cython-defined class, pylance doesn't handle them.
I really appreciate it if developers take it into account.
Thank you in advance.

[judej]

@munechika-koyo, thanks for the report. Which class/lib are you using? and could you give us a a code snippet that demonstrates the problem please?
thanks

[munechika-koyo]

I use the Raysect library which is the ray-tracing tool, especially for the plasma analysis.
This library is mostly written in cython.

In the case of using Microsoft as a python language server, the docstring popup looks like below:

In contrast to the above, in case the of using Pylance:

Both of them are shown when the cursor hovers on the World class implemented in Raysect.

Here, I put some environmental information.

Environment data

• VS Code version: 1.51.1
• Pylance extension version: v2020.12.0
• OS and version: Windows 10, and I use Remote-WSL.
• Python version: 3.6.10 64-bit in Anaconda env.

[jakebailey]

Thanks for the info. Yes, this appears to be compiled via cython. MPLS scraped every compiled module, Pylance does not.

[cnaccio]

Just noticed this issue as well using the Zipline library which uses cython throughout the project. Switched to the Microsoft language server for now, but def prefer Pylance.

[polm]

Just a note that this is the same as #166, which is the oldest / most active topic on the issue of Cython support.

EDIT: Ah, actually #166 is a bit more general than Cython, though still on the same topic.

[judej]

Closing this as we have no plans to scrape compiled modules in Pylance

[alexreg]

@judej It's understandable that you may not wish to tackle this issue immediately (due to limited "bandwidth") — but it's also very disappointing, considering how widespread Cython is. Wouldn't it be better simply to triage this?

Also, I don't think you necessarily need to "scrape compiled modules" to offer Cython support.

[erictraut]

I think the right answer here is for Cython toolchain to add support for generating stub files with docstrings. Those stubs can then be distributed along with the library. Any other approach I can think of (including scraping based on runtime reflection) will produce results that leave significant gaps in terms of completeness and correctness. The maintainers of Cython are in a much better place to do this work than the Pylance team. We are happy to work with the Cython community in an advisory role.

[alexreg]

@erictraut Agreed, that approach makes a lot of sense. I'll create a new issue on the Cython repo, to make the developers there aware of things.

[scoder]

I think the right answer here is for Cython toolchain to add support for generating stub files with docstrings. Those stubs can then be distributed along with the library. Any other approach I can think of (including scraping based on runtime reflection) will produce results that leave significant gaps in terms of completeness and correctness.

@erictraut, could you elaborate on this? Bothering users with having to generate and distribute additional stub files seems really annoying for everyone. What type information do you get from source code and stubs that is not available from the normal Python runtime introspection? Isn't runtime information also cheaper and more universal to collect than stub files that a) have to be made available by someone, b) have to be available where they are used and c) have to be parsed as source code?

[erictraut]

Yes, users who consume a library should not need to discover, download, or generate stub files. Type information should always be included in a library package and downloaded and installed as part of the library package. For libraries that are generated entirely or partly using the Cython compiler, Cython could automatically generate these stubs. A library author could then include the stubs alongside the compiled binaries within their package.

Runtime information obtained through introspection of a compiled module is insufficient for a number of reasons. First, accessing runtime type information requires loading and running the code in the library. As a rule, static tools like Pylance never load or run code. Doing so would be a major security problem. All type information used by Pylance must be in a declarative form — type annotations that are inlined in Python source (".py") files or included in stub (".pyi") files. Second, runtime type information is incomplete. It doesn't include function parameter or return types. These types are known by the Cython compiler, so it should be able to emit stubs that include this type information.

Here's documentation we've created for library authors that discusses the type information that should be included in a library if they would like it to work well with Pylance and other static analysis tools (language servers, type checkers, linters, etc.).

[scoder]

Runtime information obtained through introspection of a compiled module is insufficient for a number of reasons. First, accessing runtime type information requires loading and running the code in the library. As a rule, static tools like Pylance never load or run code.

I understand that argument. It's also a dedicated goal of PEP-484 & friends.

Doing so would be a major security problem.

Assuming that we are talking about IDEs, AFAIR Python's setuptools has autoimport features, so any Python package that I install in my venv can execute code when I launch my Python runtime. Or can overwrite other packages with its own code that I would commonly import. Whether a static analysis tool can also trigger code doesn't seem to add anything substantial to that threat.

Regardless, this is the wrong place to discuss something like this (yet again). I'd just like to see people stop claiming that importing a module is a security threat, while installing software from untrusted sources is the actual threat.

Second, runtime type information is incomplete. It doesn't include function parameter or return types.

That's what __annotations__ is for. These dicts provide strings in Cython that can be parsed as type information.

And it looks like users have exploited this before to generate stub files from Python annotations: python/mypy#7542
That approach (stubgen) seems generally helpful.

I think the main issue is that special methods in CPython cannot expose their signature. And Cython doesn't (currently?) have a way to work around that. But the only real issues here are with __init__, __new__ and __call__, since all other special methods have a fixed signature. So, the missing part of the type information is overall very limited.

These types are known by the Cython compiler, so it should be able to emit stubs that include this type information.

I agree, see cython/cython#3150
Regardless, since stubgen exists, it would probably get users pretty far already to use it.

[JMPZ11]

Forgive the french, but closing this is horse shit. Python's c(++) interop is a selling point, and pylance not supporting Cylance is a HUGE miss; especially considering we know it can be done ala MPLS.

Fix this please.
Thanks

[FeldrinH]

@erictraut Agreed, that approach makes a lot of sense. I'll create a new issue on the Cython repo, to make the developers there aware of things.

@alexreg Did you create that issue? I looked at the Cython repo issue tracker and couldn't find a relevant issue, unless you count cython/cython#3150, but the original request there seems to be about a different thing.

[scoder]

cython/cython#3150 is the right ticket.