Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support PEP-0484 .pyi stub files #839

Closed
patrys opened this issue Jan 25, 2017 · 40 comments

Comments

@patrys
Copy link

commented Jan 25, 2017

These are not currently supported but it would be useful to have them.

Support is tricky as Jedi seems to do definition discovery and type discover in the same parsing/evaluation pass. My initial ideas was to load a .pyi file instead of the .py file when looking for a module and it seems to work for type inference but can't possibly work for providing actual definitions as in the stub all of the definitions are dummy ellipsis objects (which means names match but things like offsets won't).

I think it's safe to follow what mypy does and ignore the .py file entirely when doing completion if a .pyi file exists but it's unclear to me how to decouple completion from the definition provider (go to assignment, go to definition etc.).

@davidhalter

This comment has been minimized.

Copy link
Owner

commented Jan 25, 2017

Good points. I'm not sure yet where we want to go with them. I'm against just ignoring the .py file, because that makes it pretty much impossible to use goto_definitions properly. I would probably just add a hook into the function execution logic of Jedi (and maybe one for param inference). That's the biggest problem AFAIK.

Support is tricky as Jedi seems to do definition discovery and type discover in the same parsing/evaluation pass.

What do you mean by that? By the way, I've almost completely rewritten type inference the dev branch. Just saying in case you have looked at master.

@patrys

This comment has been minimized.

Copy link
Author

commented Jan 25, 2017

My initial idea was to separate the definition lookup from type lookup.

Type lookup could theoretically parse just the .pyi files where available and otherwise use the current parser/evaluator to find where all imports come from.

Definition lookup could theoretically reuse completion info to find which module a particular symbol comes from and only re-parse that single file if all completion had was a stub.

Tricky stuff happens when completions are needed within the same file. I'm not sure what PyCharm does here: should it assume that the stub is up to date and use that possibly missing a signature for a newly defined function? Should it ignore the stub entirely and rely on the evaluator to give it a correct signature? Should we somehow combine both of these relying on symbol names matching?

@patrys

This comment has been minimized.

Copy link
Author

commented Jan 25, 2017

Another question is what to do with type signature comments such as # type: (int, int) -> str. If I wanted to have them supported, should that be up to the parser (I think that's how Python 3 type annotations are handled) or evaluator? I think these share a fair amount of scope with stub files: they provide an alternative source of signatures for functions and classes that Jedi possibly already has an (in)complete signature based on the inference engine.

@davidhalter

This comment has been minimized.

Copy link
Owner

commented Jan 25, 2017

Are you planning to implement this feature yourself? I'm happy to help you with those questions, because they are really good. However if you're not planning to add this feature, I'll leave them open for a future discussion.

@patrys

This comment has been minimized.

Copy link
Author

commented Jan 25, 2017

I'm happy to work on this as I've already worked on a number of tools using Jedi's API (as contributions to plugins for Sublime, Atom and most recently VSCode). I would obviously need your guidance but in exchange would be happy to contribute more docstrings for Jedi internals.

@patrys

This comment has been minimized.

Copy link
Author

commented Jan 26, 2017

Another question: how to store overloaded signatures?

@overload
def foo(a: int) -> str: ...
@overload
def foo(a: str) -> None: ...
@davidhalter

This comment has been minimized.

Copy link
Owner

commented Jan 28, 2017

You need to give me a bit of time, I need to work out a plan first (after the next release).

@davidhalter

This comment has been minimized.

Copy link
Owner

commented Feb 5, 2017

Ok. I suggest we ignore the issue with overload for now. I think that is something that we have to track separately. (Because it may appear in normal files as well?!)

There's a check that looks for the pep 484 parameters when Jedi executes functions. I think for return values this is good enough. For params there's a similar check I think. At both of these places we could just try to lookup a file that has the .pyi name.

I'm not exactly sure where we would need to search for those files.

I think one remaining though would be: It shouldn't slow down Jedi too much.

Let me know if you need to know more.

@gopar

This comment has been minimized.

Copy link

commented Oct 23, 2017

Any updates on this?

@lazka

This comment has been minimized.

Copy link

commented Mar 1, 2018

I've added stubs to pycairo recently (the latest release on pypi): https://github.com/pygobject/pycairo/blob/master/cairo/__init__.pyi if someone needs a package to test with

@tristan957

This comment has been minimized.

Copy link

commented Mar 22, 2018

@lazka thank you for your GTK documentation. It is amazing.

@idchlife

This comment has been minimized.

Copy link

commented Jul 25, 2018

Wow! Jedi would support .pyi stubs files? This is insanely wonderful.
Seeing how TypeScript ecosystem benefits from actual .ts files and .d.ts (only definition files, adding or replacing type definitions for existing symbols, modules) I imagine that jedi support for .pyi files will fire interest for using and writing types in python community and we will end up with huge typeshed with many types, like typescript has DefinitelyTyped.

Edit: there will be a problem as I can understand from OP post with .py files of existing module and .pyi files of stubs.
So actually there will be less benefit of incomplete .pyi files than lack of them completely, yes?
(if .pyi will not have new variable and it will be inside "real" .py file - jedi will not show new variable in .py file)

@Goom11

This comment has been minimized.

Copy link

commented Mar 5, 2019

Any updates on this? I would greatly benefit from support for .pyi files. I also have the use case where I have folders with only .pyi files. So go to definition would hopefully go to the .pyi files themselves as there is no .py file to go to.

@davidhalter

This comment has been minimized.

Copy link
Owner

commented Mar 6, 2019

We're actually pretty far on this. Please try the typeshed branch and let me know if there are still issues with it.

Once the typeshed branch is working, we can get stubs working in general, because stdlib stubs are already working on that branch. This means that they should also be working in general, only the lookup is missing on PYTHONPATH.

So I would say this is 80% done, just needs a bit more effort and a merge to master.

@Goom11

This comment has been minimized.

Copy link

commented Mar 7, 2019

@davidhalter, I verified that I am able to go to definition to .pyi files in stdlib.

I used a simple script.

import asyncio
async def call():
    pass
loop = asyncio.get_event_loop()
loop.run_until_complete(call())
loop.close()

If I go-to-definition on the get_event_loop, it takes me to asyncio/events.py.
If I go-to-definition on the run_until_complete, it takes me to .../stdlib/3/asyncio/events.pyi.

Is this expected behavior or should the go-to-definition on run_until_complete take me to run_until_complete in AbstractEventLoop in asyncio/events.py? Without the typeshed branch, go-to-definition doesn't do anything for run_until_complete. Same for close. Nonetheless, this is likely unrelated to this issue and this behavior is an awesome improvement.

Thanks so much! Let me know if I can do anything to add support for general stubs as that is my primary use case.

@Goom11

This comment has been minimized.

Copy link

commented Mar 7, 2019

In fact, I would appreciate a code pointer for adding support for general stubs.

@davidhalter

This comment has been minimized.

Copy link
Owner

commented Mar 10, 2019

IMO stubs are not there to get you to the stubs (for goto/completions), but to assist type inference and for builtins. So with that branch you should be able to goto the stub of math.cos, but not to os.walk, because you're mostly going to be more interested in the implementation than its types. However, the result of os.walk() should be the result of the stub.

It might at the moment not be possible to use goto properly in all Jedi plugins with the typeshed branch. I have therefore started davidhalter/jedi-vim#915. If you want to test stubs you can checkout the typeshed branch and copy your stubs to jedi/third_party/typeshed/third_party/2and3.

@patrys

This comment has been minimized.

Copy link
Author

commented Mar 10, 2019

The Language Server protocol offers both going to definition and to declaration. Typescript server implements both and both are useful as sometimes you want to dig deeper into the type system.

@davidhalter

This comment has been minimized.

Copy link
Owner

commented Mar 10, 2019

Interesting. I'd be fine with releasing both, it's like an hour of work only anyway.

However what I don't really understand is the difference of the following:

  • Goto Declaration Request
  • Goto Definition Request
  • Goto Type Definition Request
  • Goto Implementation Request

I'm not 100% sure what these are from reading the doucmentation on https://microsoft.github.io/language-server-protocol/specification#textDocument_declaration. Can you help me understand the exact differences and what they would be for Python?

@patrys

This comment has been minimized.

Copy link
Author

commented Mar 10, 2019

Sorry, I confused declaration with type definition. Typescript extension actually provides those two where "go to definition" opens a .js file and "go to type definition" the corresponding .d.ts file (the Typescript counterpart of .pyi files) if case of code with external typing. If types are inline, both open the .ts file.

I guess declaration/definition distinction is useful for languages with either header files or where public APIs need to be declared explicitly (like Pascal "units").

Implementation request is for code "specializing" the symbol at hand (extending an abstract base class, implementing a virtual method, implementing the highlighted interface etc.). I think for Python it would be useful to get a list of locations sub-classing the highlighted class or overloading the highlighted method. I'm not sure there's any documentation that describes this in depth other than discussions around the LSP spec mentioning some intended uses or examples of application.

@Goom11

This comment has been minimized.

Copy link

commented Mar 11, 2019

@davidhalter, could we support stubs for goto_definitions / goto_assignments & autocomplete?

For example, suppose there is a file, helper.pyi, with the contents:

def greeting(name: str) -> str: ...

and main.py with the contents:

import helper
x = helper.greeting("hello")

Facebook Thrift (https://github.com/facebook/fbthrift) is an example of a use case where it generates .pyi files for ease of development.

  1. I would like for completions on helper. to provide greeting.
  2. I would like for goto_definitions / goto_assignments on the helper and greeting and x to provide something as well.

I'm willing to help in whatever way I can.

cc: @mostafaeweda or @kcaze may be able to answer questions about the differences between the LSP commands.

@davidhalter

This comment has been minimized.

Copy link
Owner

commented Mar 12, 2019

@Goom11 Please try stuff like import os; os.walk on the typeshed branch. If that works, it will also work for other stubs. Currently only the stdlib is supported.

If anything doesn't work for you, please let me know. I'm happy to improve some things.

@davidhalter

This comment has been minimized.

Copy link
Owner

commented Apr 9, 2019

How do you guys want to use stubs? Like foo.py with a foo.pyi file right next to it? I will implement that if that's all. I feel like in PEP 561 there are other approaches as well. Are those all needed for you guys? You may speak now, otherwise it might not make the next release. :)

BTW: @patrys I'm almost done with supporting both goto declaration and goto definition. Still some work is needed, but I'm pretty far.

@patrys

This comment has been minimized.

Copy link
Author

commented Apr 9, 2019

Two use cases for me:

  1. Being able to use .pyi files that came with packages (or with a typeshed project) to tell me whether I use my imports correctly.
  2. Being able to validate my own implementation against the .pyi stub sitting next to it (so I can distribute both).
@davidhalter

This comment has been minimized.

Copy link
Owner

commented Apr 9, 2019

So for your use case (1), where are the files? Can you give me an example of the original library and the stubs? If possible also with the github links. I'm still not 100% sure how these work and I would like to have examples (that I can test against).

Typeshed is of course included by installing Jedi.

@tonyfinn

This comment has been minimized.

Copy link

commented Apr 9, 2019

@The-Compiler

This comment has been minimized.

Copy link

commented Apr 10, 2019

There are two usecases for me:

  • Improving what jedi does based on the typeshed project (though at least for the stdlib, I'm not sure if there is much to be improved)
  • Using PEP 561-like stub packages (in my case, PyQt5-stubs) for type information for C/C++ extensions.
@Goom11

This comment has been minimized.

Copy link

commented Apr 10, 2019

My use case is similar to @The-Compiler 's PyQt5-stubs use case. My use case is using PEP 561 stub-only packages for definitions and completions. These .pyi files would not be included in Typeshed but rather the location of the library would be passed via the sys_path argument to jedi.Script.

@hoefling

This comment has been minimized.

Copy link
Contributor

commented Apr 25, 2019

My use case is foo.pyi stub right next to a foo.cpython-36m-x86_64-linux-gnu.so extension (or foo.cp36-win_amd64.pyd on Windows if that matters), would it be also covered by the impl?

@Goom11

This comment has been minimized.

Copy link

commented Apr 25, 2019

I actually also have the same / similar use case as @hoefling . I have a foo.pyi file next to a foo.so file.

@davidhalter

This comment has been minimized.

Copy link
Owner

commented May 2, 2019

I guess we can cover your cases as well @hoefling and @Goom11.

Do you have docstrings in any of those files?

Or even more generally: Does someone write docstrings in they stubs?

@fredrikaverpil

This comment has been minimized.

Copy link

commented May 3, 2019

There's a shim called Qt.py which selects either PySide, PySide2, PyQt4 or PyQt5 depending on what is available. This is used as so:

import sys
from Qt import QtWidgets

app = QtWidgets.QApplication(sys.argv)
button = QtWidgets.QPushButton("Hello World")
button.show()
app.exec_()

Because of how Qt.py works, autocompletion/intellisense won't work properly of the selected Qt binding but it could work if sidecar stub files were used.

I tried these stub files (modified PyQt5 stubs to work with Qt.py) and they worked great in PyCharm, but not in vscode. When contacting the vscode devs, they said this was because stubs were not supported by jedi.

@davidhalter Would your additions possibly make this work better now in vscode?


@ctismer @cmaureir Are there any stubs bundled with PySide2 which could potentially be used out of the box?
EDIT: awesome, seems the .pyi files for PySide2 is located under venv/lib/python3.7/site-packages/PySide2/*.pyi

@davidhalter

This comment has been minimized.

Copy link
Owner

commented May 3, 2019

@fredrikaverpil Not sure if Qt.py is going to work initially, but if it doesn't you can create a separate issue. I'm pretty sure the fix is trivial once PySide2 stubs are working.

@fredrikaverpil

This comment has been minimized.

Copy link

commented May 3, 2019

@davidhalter I am guessing I need to be running a different version of jedi inside vscode (than the one bundled) to test this?
I created, maybe prematurely, an issue here: #1318

PySide2 stubs (via Qt.py imports) works fine inside PyCharm with PySide 5.12.3.

@hoefling

This comment has been minimized.

Copy link
Contributor

commented May 3, 2019

@davidhalter indeed, I include docstrings in stubs; I use them to generate Sphinx docs for C extension modules with a custom Sphinx extension similar to this one. Not sure if it's a common usecase though.

@cmaureir

This comment has been minimized.

Copy link

commented May 4, 2019

late to the party, but yeah, we are shipping .pyi files since a couple of versions ago. If you find any issue with them, let us know!

@davidhalter

This comment has been minimized.

Copy link
Owner

commented May 19, 2019

The whole typeshed/stubs support is now on master. Please test and report errors. Release coming soon.

@davidhalter

This comment has been minimized.

Copy link
Owner

commented Jun 20, 2019

v0.14.0 is released 🎆 In the end stubs were way more work than I originally thought - I ended up rewriting all "PEP 484" related things. It took a year and was probably like 3-4 months of work in 8 hour work days.

Please test and report bugs.

I hope all your requirements are working. Please let me know if they are not. I kind of lost track about all the different use cases.

BTW I usually announce releases on twitter as well: https://twitter.com/jedidjah_ch (no spam to be expected)

@hoefling

This comment has been minimized.

Copy link
Contributor

commented Jun 20, 2019

@davidhalter ran my tests now with 0.14 and my workarounds (ugly stuff like monkeypatching jedi.evaluate.compiled.load_module) removed and It Just Works™. This is so awesome. Thank you for the release!

Looking at the code changes, it's hard to miss a new plugin system - are you planning on making it public, possibly with an easier plugging support (e.g. setuptools entrypoints group)? I find it incredibly useful if I can now put my own stuff (like various objects counting to infer probabilities for each completion) in a plugin.

@davidhalter

This comment has been minimized.

Copy link
Owner

commented Jun 20, 2019

Yeah, there's a plugin system that you may hijack if you want, but the API is just not stable at the moment and may change at any time.

I generally intend to release a plugin API at some point, but I'm not really sure, yet what people would use it for. I don't really understand your issues. What would you want to use it for?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
You can’t perform that action at this time.