Import Resolution issue

[lancewl]

Hi there,

I use the exact same way to create python venv in every projects. In some projects pyright is able to find those site-packages under venv folder, but some projects just doesn't work.

I did some experiments to see if I can find any pattern that why it works occasionally. Here is the steps I take:

mkdir tmp
cd tmp
echo "import requests" >> test.py
python -m venv .venv
source .venv/bin/activate
pip install requests
pyright test.py --verbose

No configuration file found.
No pyproject.toml file found.
stubPath /Users/lancewenglin/Dev/tmp/typings is not a valid directory.
Assuming Python platform Darwin
Search paths for /Users/lancewenglin/Dev/tmp
  /Users/lancewenglin/.npm/_npx/d494279c469adf60/node_modules/pyright/dist/typeshed-fallback/stdlib
  /Users/lancewenglin/Dev/tmp
  /Users/lancewenglin/Dev/tmp/typings
  /Users/lancewenglin/.npm/_npx/d494279c469adf60/node_modules/pyright/dist/typeshed-fallback/stubs/...
  /Users/lancewenglin/.pyenv/versions/3.7.12/lib/python3.7
  /Users/lancewenglin/.pyenv/versions/3.7.12/lib/python3.7/lib-dynload
  /Users/lancewenglin/.pyenv/versions/3.7.12/lib/python3.7/site-packages
Searching for source files
Found 1 source file
pyright 1.1.265
/Users/lancewenglin/Dev/tmp/test.py
  /Users/lancewenglin/Dev/tmp/test.py:1:8 - warning: Import "requests" could not be resolved from source (reportMissingModuleSource)
0 errors, 1 warning, 0 informations
Completed in 0.467sec

From the logs, looks like pyright cannot find the venv packages.

However, I tried to source the venv from the other projects that has installed requests package as well:

deactivate
source ../project_A/.venv/bin/activate
pyright test.py --verbose

No configuration file found.
No pyproject.toml file found.
stubPath /Users/lancewenglin/Dev/tmp/typings is not a valid directory.
Assuming Python platform Darwin
Search paths for /Users/lancewenglin/Dev/tmp
  /Users/lancewenglin/.npm/_npx/d494279c469adf60/node_modules/pyright/dist/typeshed-fallback/stdlib
  /Users/lancewenglin/Dev/tmp
  /Users/lancewenglin/Dev/tmp/typings
  /Users/lancewenglin/.npm/_npx/d494279c469adf60/node_modules/pyright/dist/typeshed-fallback/stubs/...
  /Users/lancewenglin/.pyenv/versions/3.7.12/lib/python3.7
  /Users/lancewenglin/.pyenv/versions/3.7.12/lib/python3.7/lib-dynload
  /Users/lancewenglin/Dev/project_A/.venv/lib/python3.7/site-packages
  /Users/lancewenglin/Dev/project_A/src
  /Users/lancewenglin/Dev/project_A/serverless_utils/python_packages/canoo/python_utils/src
Searching for source files
Found 1 source file
pyright 1.1.265
0 errors, 0 warnings, 0 informations
Completed in 0.454sec

pyright works perfectly with venv.

The only pattern I found is that every old venv I created before works fine, and every newly created venv doesn't work. I've been struggling in this for days.

Any ideas or suggestions are very appreciate!!!

[erictraut]

I'm not able to repro the problem as reported. I wonder if you have a corrupted Python install or a pip cache?

After the failure, have you looked in the site-packages directory within your newly-created .venv? Is there a requires directory with an __init__.py file within it?

It looks like you're using Python 3.7. Do you happen to have any other version installed? If so, perhaps you can try your repro steps using a different interpreter version, like python3.8 -m venv .venv. Are you able to repro it then?

[lancewl]

Thanks for getting back to me!

Yeah, there is a .venv/lib/python3.7/site-packages/requests/__init__.py file.

I've switched to python 3.8.13 by using pyenv and tried the steps again, but no luck.

[erictraut]

When you say "no luck", do you mean that you can't repro the problem with 3.8.13? Or you're still seeing the error?

[lancewl]

Still the same error with 3.8.13. Pyright won't check site-packages under .venv.

[erictraut]

Sorry, but I don't have any good ideas here. I'm not able to repro the problem, and I don't have any good theories other than that there is something corrupt on your system.

Do you have another computer or virtual machine where you could try to repro? Perhaps you can explore the site-packages directory your .venv and see if you find anything that looks suspicious.

If you open the project with VS Code (with Pylance or Pyright installed), do you see the same error in the IDE? If so, when you right-click on the error and choose "Go To Definition", where does it take you?

[lancewl]

I just tried on another laptop(MacBook) with python 3.9.7(installed via pyenv), and I'm having the same error as well. It doesn't check the .venv folder. I tried to check the directory in the .venv folder and compare with those folders that are working, but I can't find any difference.

I'm not using VS Code, but I can give it a try later!

If I directly call python and try to import the library, it works fine in the python console. I wonder if there is any cases that pyright will ignore the folders when looking for site-packages?

Thanks again for looking into this. I'll keep trying different methods to see if I can figure out a pattern.

[erictraut]

Are you sure that you're activating the local venv before running pyright? When pyright runs, it shell executes python3 and runs a small script that returns the contents of sys.path. The resulting list of paths are then used for import resolution.

[lancewl]

I'm pretty sure that I've activated my venv before running pyright. I tried to print out the sys.path from python3, and here is the output:

Python 3.9.7 (default, Sep 12 2021, 23:23:43)
[Clang 12.0.5 (clang-1205.0.22.11)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path
['', '/Users/lance/.pyenv/versions/3.9.7/lib/python39.zip', '/Users/lance/.pyenv/versions/3.9.7/lib/python3.9', '/Users/lance/.pyenv/versions/3.9.7/lib/python3.9/lib-dynload', '/Users/lance/Dev/test/.venv/lib/python3.9/site-packages']
>>>

Looks like python3 is able to find site-packages in .venv

[erictraut]

It looks like there's a discrepancy here. In the log from your earlier post, your search paths were configured to point to /Users/lancewenglin/.pyenv/versions/3.7.12/lib/python3.7. In your most recent post, the search paths point to /Users/lance/Dev/test/.venv/lib/python3.9.

I think this is strong evidence that when you were running pyright earlier you hadn't activated your local venv, and that would explain the diagnostic output.

[lancewl]

The reason why those paths are different is that they were running from different laptop, and the earliest log is from pyrgiht, the latest one is from python3.

I've go through my testing steps again, and here is the full output, hope that this can explain what I'm facing right now:

~/Dev/test via 🐍 3.9.7
➤ source .venv/bin/activate

~/Dev/test (test) via 🐍 3.9.7
➤ which python
/Users/lance/Dev/test/.venv/bin/python

~/Dev/test (test) via 🐍 3.9.7
➤ which python3
/Users/lance/Dev/test/.venv/bin/python3

~/Dev/test (test) via 🐍 3.9.7
➤ python3
Python 3.9.7 (default, Sep 12 2021, 23:23:43)
[Clang 12.0.5 (clang-1205.0.22.11)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path
['', '/Users/lance/.pyenv/versions/3.9.7/lib/python39.zip', '/Users/lance/.pyenv/versions/3.9.7/lib/python3.9', '/Users/lance/.pyenv/versions/3.9.7/lib/python3.9/lib-dynload', '/Users/lance/Dev/test/.venv/lib/python3.9/site-packages']
>>> exit()

~/Dev/test (test) via 🐍 3.9.7 took 6m 50s
➤ pyright test.py --verbose
No configuration file found.
No pyproject.toml file found.
stubPath /Users/lance/Dev/test/typings is not a valid directory.
Assuming Python platform Darwin
Search paths for /Users/lance/Dev/test
  /Users/lance/.npm/_npx/d494279c469adf60/node_modules/pyright/dist/typeshed-fallback/stdlib
  /Users/lance/Dev/test
  /Users/lance/Dev/test/typings
  /Users/lance/.npm/_npx/d494279c469adf60/node_modules/pyright/dist/typeshed-fallback/stubs/...
  /Users/lance/.pyenv/versions/3.9.7/lib/python3.9
  /Users/lance/.pyenv/versions/3.9.7/lib/python3.9/lib-dynload
  /Users/lance/.pyenv/versions/3.9.7/lib/python3.9/site-packages
Searching for source files
Found 1 source file
pyright 1.1.265
/Users/lance/Dev/test/test.py
  /Users/lance/Dev/test/test.py:1:8 - warning: Import "requests" could not be resolved from source (reportMissingModuleSource)
0 errors, 1 warning, 0 informations
Completed in 0.637sec

~/Dev/test (test) via 🐍 3.9.7 took 3s
➤

From the above logs, you can see that both python and python3 are linked to the right location(venv), and also when I try to print out sys.path directly in python3, it shows venv path, but when I run pyright, it cannot find venv paths.

[erictraut]

Thanks for the additional information. Unfortunately, I'm still not able to repro it. We have extensive verbose logging in place when an import resolution fails completely, but we don't have similar logging in place for when the import resolution succeeds in finding a type stub but fails to find the corresponding source file. I've added additional verbose logging, and this will be included in the next release of pyright. That should help us with additional clues about why you're seeing this problem.

[erictraut]

The additional logging is included in pyright 1.1.266, which I just published. Please repro with the --verbose flag and post the resulting log output.

[lancewl]

Thanks for your help again! Here is the new output:

➤ pyright test.py --verbose
No configuration file found.
No pyproject.toml file found.
stubPath /Users/lance/Dev/test/typings is not a valid directory.
Assuming Python platform Darwin
Search paths for /Users/lance/Dev/test
  /Users/lance/.npm/_npx/898d6ca37da84b66/node_modules/pyright/dist/typeshed-fallback/stdlib
  /Users/lance/Dev/test
  /Users/lance/Dev/test/typings
  /Users/lance/.npm/_npx/898d6ca37da84b66/node_modules/pyright/dist/typeshed-fallback/stubs/...
  /Users/lance/.pyenv/versions/3.9.7/lib/python3.9
  /Users/lance/.pyenv/versions/3.9.7/lib/python3.9/lib-dynload
  /Users/lance/.pyenv/versions/3.9.7/lib/python3.9/site-packages
Searching for source files
Found 1 source file
Could not resolve source for 'requests' in file '/Users/lance/Dev/test/test.py'
  Looking in root directory of execution environment '/Users/lance/Dev/test'
  Attempting to resolve using root path '/Users/lance/Dev/test'
  Looking in python search path '/Users/lance/.pyenv/versions/3.9.7/lib/python3.9'
  Attempting to resolve using root path '/Users/lance/.pyenv/versions/3.9.7/lib/python3.9'
  Looking in python search path '/Users/lance/.pyenv/versions/3.9.7/lib/python3.9/lib-dynload'
  Attempting to resolve using root path '/Users/lance/.pyenv/versions/3.9.7/lib/python3.9/lib-dynload'
  Looking in python search path '/Users/lance/.pyenv/versions/3.9.7/lib/python3.9/site-packages'
  Attempting to resolve using root path '/Users/lance/.pyenv/versions/3.9.7/lib/python3.9/site-packages'
pyright 1.1.266
/Users/lance/Dev/test/test.py
  /Users/lance/Dev/test/test.py:1:8 - warning: Import "requests" could not be resolved from source (reportMissingModuleSource)
0 errors, 1 warning, 0 informations
Completed in 0.665sec

[erictraut]

If you look in the directory '/Users/lance/.pyenv/versions/3.9.7/lib/python3.9/site-packages', do you see a subdirectory named requests and a file within that named __init__.py?

[lancewl]

Nope, I didn't install requests with pyenv's python. I installed requests within venv.

[erictraut]

The site-packages directory for .venv isn't one of the search paths that pyright is considering. Let's try to determine why. When you manually examined sys.path, it was present.

Here's the full script that pyright executes to extract the sys.path search paths from the currently-configured Python environment.

import os, os.path, sys
normalize = lambda p: os.path.normcase(os.path.normpath(p))
cwd = normalize(os.getcwd())
orig_sys_path = [p for p in sys.path if p != ""]
sys.path[:] = [p for p in sys.path if p != "" and normalize(p) != cwd]
import sys, json
json.dump(dict(path=orig_sys_path, prefix=sys.prefix), sys.stdout)

Please paste this code into temp source file, then switch to the root directory of your project, and execute the script using python3 <path of script>. Does the output of the script include '/Users/lance/Dev/test/.venv/lib/python3.9/site-packages'?

[lancewl]

Yeah the script print out .venv path. Here is the output:

~/Dev/test (test) via 🐍 3.9.7 took 5s
➤ python3 path.py
{"path": ["/Users/lance/Dev/test", "/Users/lance/.pyenv/versions/3.9.7/lib/python39.zip", "/Users/lance/.pyenv/versions/3.9.7/lib/python3.9", "/Users/lance/.pyenv/versions/3.9.7/lib/python3.9/lib-dynload", "/Users/lance/Dev/test/.venv/lib/python3.9/site-packages"], "prefix": "/Users/lance/Dev/test/.venv"}%

[erictraut]

Very strange. And you're sure you activated this venv prior to running the script?

I'm looking through the pyright code to see if I can come up with any other theories that might explain what you're seeing. The code first attempts to execute the script using python3 <script>. If it receives an exception when executing this script, it tries again using python <script> (i.e. it removes the "3" from python3). Could you try running the script using python instead of python3 to see if that might explain what you're seeing?

You're the only person who has reported this issue. If this were a bug in pyright, I would have expected to have seen this issue reported before. That leads me to hypothesize that it's something specific to your system or the way it's configured.

[lancewl]

Sorry for the late reply. I was busy at work today. I just run the script with python and get the exact same output. I've also double check that I had activated venv before I run the script.

Yeah I agree that it might be something wrong on my machine, but I've tried a lot of different methods, and even switch laptop, but still can't get any ideas on why it is not working.

I'm using zsh on both laptops. Let me try different shell.

[lancewl]

I tried using bash, and it works perfectly. Hmmm so we can narrow down the issue into only zsh now. Let me dig into my zshrc to see if there's something that cause the issue. Although we can ensure that this only happens in my zsh, I think it still can not explain why the sys.path works in zsh and pyright isn't.

[erictraut]

Pyright is using node's execFileSync to do a shell execute. Here's the documentation.

OK, it's good to know this is an issue with zsh. You're the first person who has ever run into this, which indicates to me that very few people are using zsh. (I have to admit that I had never heard of it before.)

I don't think there's anything pyright can do differently here, so I think you'll need to either switch to a compatible shell or find some workaround for zsh.

[abdulhaq-e]

I use zsh and pyright works just fine. So it's probably the shell configuration for @lancewl

@erictraut:

which indicates to me that very few people are using zsh.

I expect a high percentage of pyright users who use Linux/MacOS will be using zsh.

Thanks for all your work, much appreciated!

[erictraut]

Got it. Thanks for the clarification.

[lancewl]

Thanks again @erictraut! I will figure out what's run on my zsh configuration, and update my solution here!