Go to Definition always opens up stub .pyi file even .py file exists

[copdips]

• Language Server version: 2024.6.1
• OS and version: Windows subsystem for Linux - Ubuntu 20.04
• Python version (& distribution if applicable, e.g. Anaconda): 3.11

Code Snippet

Hello,

go-to-definition never works as long as .pyi files exists.

For example, go-to-definition over getLogger in the below example always opens up pyalnce's stub instead of orignal python file:
/home/xiang/.vscode-server/extensions/ms-python.vscode-pylance-2024.6.1/dist/typeshed-fallback/stdlib/logging/__init__.pyi

I expect it to open up the real source code:
/home/xiang/opt/python3.11/lib/python3.11/logging/__init__.py

import logging
logging.getLogger("foo")

If I switch the user settings to "python.languageServer": "Jedi", then go-to-definition works, but unfortunately, I lost all the Pylance's features.

[heejaechang]

I just tried and it worked as expected with latest pylance prerelease bit.

Can you provide us some logs as described in the troubleshooting guide?

[KacieKK]

I cant repro, either; tried 6.1 & 6.100 same results

[bschnurr]

maybe double check you've selected to correct python environment

[copdips]

hello @bschnurr,

the python interpreter should be correct, I'm using the .venv folder under the root of the workspace:

@heejaechang

please find attached pylance logs:
pylance_language_server_output.txt

pylance_2024.6.1_id_0.txt

my test file is : /home/xiang/git/fastapi-demo/_local_test/pylance.py

and I'm sure I clicked on the go to definition (F12):

[heejaechang]

can we see what you get when you run sys.path in your .venv? asking since I see this from log

[Info] (8329) Search paths for file:///home/xiang/git/fastapi-demo
[Info] (8329)   /home/xiang/.vscode-server/extensions/ms-python.vscode-pylance-2024.6.1/dist/typeshed-fallback/stdlib
[Info] (8329)   /home/xiang/git/fastapi-demo
[Info] (8329)   /home/xiang/git/fastapi-demo/typings
[Info] (8329)   /home/xiang/.vscode-server/extensions/ms-python.vscode-pylance-2024.6.1/dist/typeshed-fallback/stubs/...
[Info] (8329)   /home/xiang/.vscode-server/extensions/ms-python.vscode-pylance-2024.6.1/dist/bundled/stubs
[Info] (8329)   /home/xiang/git/fastapi-demo/.venv/lib/python3.11/site-packages
[Info] (8329)   /home/xiang/git/fastapi-demo/.venv/lib64/python3.11/site-packages

missing something like this

Info] (8329)   /home/xiang/opt/python3.11/lib/python3.11
[Info] (8329)   /home/xiang/opt/python3.11/lib/python3.11/lib-dynload

which is where the source file should exist.

I see <default workspace> has it correctly, so it must be something specific to that fastapi-demo folder. you must have somewhere that rewrite sys.path.

[copdips]

(.venv 3.11) 00:18 $ which python
/home/xiang/git/fastapi-demo/.venv/bin/python

(.venv 3.11) 00:18 $ python
Python 3.11.7 (main, Jan 13 2024, 23:46:53) [GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path
['', '/home/xiang/opt/python3.11/lib/python311.zip', '/home/xiang/opt/python3.11/lib/python3.11', '/home/xiang/opt/python3.11/lib/python3.11/lib-dynload', '/home/xiang/git/fastapi-demo/.venv/lib/python3.11/site-packages']
>>> 

the file /home/xiang/git/fastapi-demo/_local_test/pylance.py only has 2 lines of code as shown above.

[copdips]

more info about the python interpreters, and one more question, the last deactivate to exit the venv failed for a permission issue, is it expected ?

(.venv 3.11) 00:29 $ ll /bin/python*
lrwxrwxrwx 1 root root       7 Apr 15  2020 /bin/python -> python2*
lrwxrwxrwx 1 root root      14 Apr 15  2020 /bin/python-config -> python2-config*
lrwxrwxrwx 1 root root       9 Mar 13  2020 /bin/python2 -> python2.7*
lrwxrwxrwx 1 root root      16 Mar 13  2020 /bin/python2-config -> python2.7-config*
-rwxr-xr-x 1 root root 3662000 Jan 31 17:23 /bin/python2.7*
lrwxrwxrwx 1 root root      33 Jan 31 17:23 /bin/python2.7-config -> x86_64-linux-gnu-python2.7-config*
lrwxrwxrwx 1 root root       9 Mar 13  2020 /bin/python3 -> python3.8*
lrwxrwxrwx 1 root root      16 Mar 13  2020 /bin/python3-config -> python3.8-config*
lrwxrwxrwx 1 root root      41 Apr  1  2022 /bin/python3.10 -> /home/xiang/opt/python3.10/bin/python3.10*
lrwxrwxrwx 1 root root      41 Jan 13 23:49 /bin/python3.11 -> /home/xiang/opt/python3.11/bin/python3.11*
lrwxrwxrwx 1 root root      41 May  1 14:10 /bin/python3.12 -> /home/xiang/opt/python3.12/bin/python3.12*
-rwxr-xr-x 1 root root 5465880 Nov 22  2023 /bin/python3.8*
lrwxrwxrwx 1 root root      33 Nov 22  2023 /bin/python3.8-config -> x86_64-linux-gnu-python3.8-config*
lrwxrwxrwx 1 root root      39 Mar 16  2021 /bin/python3.9 -> /home/xiang/opt/python3.9/bin/python3.9*

(.venv 3.11) 00:29 $ which python
/home/xiang/git/fastapi-demo/.venv/bin/python

(.venv 3.11) 00:30 $ deactivate
bash: /home/xiang/.vscode-server/extensions/ms-python.python-2024.8.1/python_files/deactivate/bash/deactivate: Permission denied

[heejaechang]

@copdips hmmm.. not sure why we didn't get these 2 paths (/home/xiang/opt/python3.11/lib/python3.11, /home/xiang/opt/python3.11/lib/python3.11/lib-dynload).

meanwhle, you can put "python.analysis.extraPath = ["/home/xiang/opt/python3.11/lib/python3.11 . "/home/xiang/opt/python3.11/lib/python3.11/lib-dynload"] to workaround the issue for now.

...

@karthiknadig have you seen the permission issue before?

I am not sure why deactivate in the vscode-python runs instead of one in .venv

[copdips]

I can see that deactivate is the one from vscode-python instead of .venv due to $PATH, but I never set this by myself, it seems that it was added automatically:

[copdips]

meanwhle, you can put "python.analysis.extraPath": ["/home/xiang/opt/python3.11/lib/python3.11", "/home/xiang/opt/python3.11/lib/python3.11/lib-dynload"] to workaround the issue for now.

Thx, this workaround works !

[copdips]

@heejaechang
although the workaround is working, could you please share more details on how pylance retrieves the search path list ?
does it perform some sort of post filtering after checking the sys.path ?

[heejaechang]

I dont believe so. I think we need to add some logging to see what we got from calling sys.path before doing anything.

...

can you set "python.analysis.logLevel": "Trace" and add this to your test file import missingModule?

make sure missingModule shows error. once it shows an error, can you share the log? I think that will show us what we got from sys.path

[copdips]

here is the log with import missingModule:

pylance_2024.6.1_id_3.txt

[copdips]

I might found the cause, but I dont know why.

In fact, under the root of my project, I have the file pyproject.toml, inside I have the pyright conf as follows:

[tool.pyright]
venvPath = "."
venv = ".venv"

removing this conf makes everyting working again.

I need this conf as explained here.

[heejaechang]

so, unfortunately, if you provide explicit location to the venv, we won't call python interpreter to get sys.path and since we don't call it, we don't know where real source for stdlib is installed (but we do know where pyi for stdlib is installed, that's why CI works).

so, probably the workaround using extraPath is only solution for your case.

[copdips]

it's weird, why specifying venv path in pyright conf prevents it from finding the the real source for stdlib ?
as venv's python interpreter shows the correct sys.path:

.venv 3.11) 00:18 $ which python
/home/xiang/git/fastapi-demo/.venv/bin/python

(.venv 3.11) 00:18 $ python
Python 3.11.7 (main, Jan 13 2024, 23:46:53) [GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> [print(p) for p in sys.path]
/home/xiang/git/fastapi-demo/.venv/bin
/home/xiang/opt/python3.11/lib/python311.zip
/home/xiang/opt/python3.11/lib/python3.11
/home/xiang/opt/python3.11/lib/python3.11/lib-dynload

/home/xiang/git/fastapi-demo/.venv/lib/python3.11/site-packages
>>> 

From my understanding in Python out of pyright, venv is to specify where the additional libs are installed, not for std libs. Or maybe the venvPath and venv names in pyright conf could be improved ?

[heejaechang]

in normal setup, sys.path returns where venv is. so when venv setting is used, assumption is either python interpreter is not available, or user doesn't want to use venv from sys.path so we don't run python interpreter in isolation to get sys.path. that's why we don't know where original source for stdlib exists.

if we run python interpreter, now we need to figure out what paths returned from sys.path we need to ignore and what not. and that gets into heuristic. so not sure whether running python interpreter to get sys.path is right answer.

that said, I think what you want is 2 different settings for CI or pre-commit and vscode from same folder? I think it might be better just make pre-commit thing to work without venv hack.

[heejaechang]

I'm wondering is it possible to make pyright to always get the search paths from sys.path whenver venv is specified or not ?

well, venv exists as a way to not call sys.path, so that sounds weird.

[copdips]

that said, I think what you want is 2 different settings for CI or pre-commit and vscode from same folder? I think it might be better just make pre-commit thing to work without venv hack.

my use case requires a slightly more nuanced approach to pre-commit due to 2 scenarios:

1/ in vscode local developement, venv is used. as explained here, I need pre-commit to validate each commit, so the venvPath and venv are specified within the pyright section of my pyproject.toml file.

2/ in CI pipeline, there's no venv involved. All the third party packages are installed directly using the CI's default python interpreter. Before running pre-commit in CI, I dynamically remove the 2 conf lines (venvPath and venv) from pyproject.toml.

[heejaechang]

how about you do it opposite way? dynamically inserting 2 conf lines for pre-commit?

[copdips]

personally I can accept some level of hack in CI pipeline, but not in normal development, as other developers should use the same thing.

[heejaechang]

then, extraPath workaround might be the only way.

[copdips]

yes but not 100%, as I suppose extraPath will return the first found source code from its search paths. If my project is on python 3.12, and in extraPath, 3.12 is added after 3.11, I think 3.11's stdlib will be returned.

That means to let it working 100%, I need to set this in the .vscode/settings of each project instead of the global one, that's is cumbersome.

[thanhph111]

@copdips Here's how I handle this issue:

1. I use Poetry as my package manager and ensure that virtual environments are used both in development and CI environments. Poetry helps manage which packages are installed in each environment.
2. I omit venv and venvPath to ensure compatibility with VSCode without needing to specify python.analysis.extraPaths.
3. For pre-commit hooks, I use local pyright:

   repos:
     - repo: local
       hooks:
         - id: pyright
           name: Pyright check
           entry: poetry run pyright  # Or `.venv/bin/pyright`
           language: system
           types_or: [python, pyi]
           require_serial: true

[copdips]

@thanhph111
Thank you for the advice. In fact, I used to use a locally installed pyright (by pip not poetry) in pre-commit, but I faced some strange issue during CI without venv as metionned here, then I switched to the repo version.