Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support path (*.pth) files in extra paths #1709

Closed
pawamoy opened this issue Aug 22, 2021 · 22 comments
Closed

Support path (*.pth) files in extra paths #1709

pawamoy opened this issue Aug 22, 2021 · 22 comments
Labels
enhancement New feature or request fixed in next version (main) A fix has been implemented and will appear in an upcoming version

Comments

@pawamoy
Copy link

pawamoy commented Aug 22, 2021

I'm using PDM to develop Python projects. It has a really nice feature where instead of installing the dependencies in __pypackages__ directly, it installs them in a cache folder, and writes .pth files pointing to them in __pypackages__. It allows deduplication across projects, which saves a tremendous amount of disk-space, and also makes installs faster.

To work with PDM+VSCode, I use this configuration: https://pdm.fming.dev/#vscode

{
  "python.autoComplete.extraPaths": ["__pypackages__/<major.minor>/lib"],
  "python.analysis.extraPaths": ["__pypackages__/<major.minor>/lib"]
}

However it seems that VSCode does not follow the .pth files, which really are just text files containing the path to the actual source file(s) on a single line:

% cat __pypackages__/3.9/lib/toml.pth 
/home/pawamoy/.cache/pdm/packages/toml-0.10.2-py2.py3-none-any/lib

% tree -L 2 /home/pawamoy/.cache/pdm/packages/toml-0.10.2-py2.py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/toml-0.10.2-py2.py3-none-any/lib
├── toml
│   ├── decoder.py
│   ├── encoder.py
│   ├── __init__.py
│   ├── ordered.py
│   ├── __pycache__
│   └── tz.py
└── toml-0.10.2.dist-info
    ├── INSTALLER
    ├── LICENSE
    ├── METADATA
    ├── RECORD
    ├── top_level.txt
    └── WHEEL

3 directories, 11 files

I'd like VSCode to follow those .pth files to actually make autoComplete and analysis extra paths work.

If I understand correctly, a .pth file can contain comments starting with #, and an arbitrary number of paths (relative or absolute) separated by new lines:

# a comment
# another comment
a/relative/path
/an/absolute/path

See https://docs.python.org/3/library/site.html.
@karthiknadig
Copy link
Member

@pawamoy which language server are you using (python.languageServer)?

@pawamoy
Copy link
Author

pawamoy commented Aug 22, 2021

"Default" is selected.

@pawamoy
Copy link
Author

pawamoy commented Aug 22, 2021

And Pylance is installed so I guess it's Pylance!

@pawamoy
Copy link
Author

pawamoy commented Aug 22, 2021

Does it mean I should reopen on https://github.com/microsoft/pylance-release instead?

@karthiknadig
Copy link
Member

I can transfer this to the pylance repo.

@karthiknadig karthiknadig transferred this issue from microsoft/vscode-python Aug 22, 2021
@jakebailey
Copy link
Member

jakebailey commented Aug 23, 2021 •
edited

Firstly, we don't currently support __pypackages__ (#1441).

We do support .pth files, but not in extraPaths. They're currently only supported when you have an interpreter selected (so Python itself executes these files via the site module, and we get them when we ask the interpreter for its paths), or limited support when using the venvPath option.

To be clear, you're looking for us to try and scan directories in extraPaths for .pth files, and then use them?

@jakebailey jakebailey added the enhancement New feature or request label Aug 23, 2021
@github-actions github-actions bot removed the triage label Aug 23, 2021
@pawamoy
Copy link
Author

pawamoy commented Aug 23, 2021

Hmmmmm. Then it's strange: it should already work, here is why.

PDM offers a command that outputs shell commands to automatically enable PEP582 for the Python interpreter.

% pdm --pep582
if [ -n "$PYTHONPATH" ]; then
    export PYTHONPATH='/home/pawamoy/.local/pipx/venvs/pdm/lib/python3.9/site-packages/pdm/pep582':$PYTHONPATH
else
    export PYTHONPATH='/home/pawamoy/.local/pipx/venvs/pdm/lib/python3.9/site-packages/pdm/pep582'
fi

My shell runs these commands at each startup. So my PYTHONPATH is always at least:

% echo $PYTHONPATH
/home/pawamoy/.local/pipx/venvs/pdm/lib/python3.9/site-packages/pdm/pep582

This folder has a sitecustomize.py, which if I understand correctly gets executed each time Python starts up. I don't understand all of it, but I guess it simply adds the right <major.minor> folder of __pypackages__ to sys.path (if __pypackages__ exists in the current working directory, or a certain number of directories above). See its code here: https://github.com/pdm-project/pdm/blob/main/pdm/pep582/sitecustomize.py

Eventually, it makes my Python interpreter able to find packages under __pypackages__ and under PDM's cache folder:

% python -c "import sys; print('\n'.join(sys.path))"

/home/pawamoy/.basher-packages/pyenv/pyenv/versions/3.9.5/lib/python39.zip
/home/pawamoy/.basher-packages/pyenv/pyenv/versions/3.9.5/lib/python3.9
/home/pawamoy/.basher-packages/pyenv/pyenv/versions/3.9.5/lib/python3.9/lib-dynload
/media/data/dev/python/duty/__pypackages__/3.9/lib
/home/pawamoy/.cache/pdm/packages/ansimarkup-1.5.0-py2.py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/appdirs-1.4.4-py2.py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/astor-0.8.1-py2.py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/attrs-21.2.0-py2.py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/autoflake-1.4-py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/bandit-1.7.0-py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/black-20.8b1-py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/certifi-2021.5.30-py2.py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/charset_normalizer-2.0.4-py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/click-8.0.1-py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/colorama-0.4.4-py2.py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/coverage-5.5-cp39-cp39-manylinux2010_x86_64/lib
/home/pawamoy/.cache/pdm/packages/darglint-1.8.0-py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/dparse-0.5.1-py3-none-any/lib
/media/data/dev/python/duty/src
/home/pawamoy/.cache/pdm/packages/execnet-1.9.0-py2.py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/failprint-0.8.0-py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/flake8-3.9.2-py2.py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/flake8_bandit-2.1.2-py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/flake8_black-0.2.3-py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/flake8_bugbear-21.4.3-py36.py37.py38-none-any/lib
/home/pawamoy/.cache/pdm/packages/flake8_builtins-1.5.3-py2.py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/flake8_comprehensions-3.5.0-py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/flake8_docstrings-1.6.0-py2.py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/flake8_plugin_utils-1.3.2-py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/flake8_polyfill-1.0.2-py2.py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/flake8_pytest_style-1.5.0-py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/flake8_string_format-0.3.0-py2.py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/flake8_tidy_imports-4.3.0-py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/flake8_variables_names-0.0.4-py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/ghp_import-2.0.1-py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/git_changelog-0.4.2-py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/gitdb-4.0.7-py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/GitPython-3.1.20-py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/idna-3.2-py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/importlib_metadata-4.6.3-py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/iniconfig-1.1.1-py2.py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/isort-5.8.0-py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/Jinja2-2.11.3-py2.py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/Markdown-3.3.4-py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/MarkupSafe-2.0.1-cp39-cp39-manylinux2010_x86_64/lib
/home/pawamoy/.cache/pdm/packages/mccabe-0.6.1-py2.py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/mergedeep-1.3.4-py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/mkdocs-1.2.2-py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/mkdocs_autorefs-0.2.1-py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/mkdocs_coverage-0.2.4-py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/mkdocs_gen_files-0.3.3-py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/mkdocs_literate_nav-0.4.0-py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/mkdocs_material-7.2.2-py2.py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/mkdocs_material_extensions-1.0.1-py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/mkdocstrings-0.15.2-py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/mypy-0.910-cp39-cp39-manylinux2010_x86_64/lib
/home/pawamoy/.cache/pdm/packages/mypy_extensions-0.4.3-py2.py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/packaging-21.0-py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/pathspec-0.9.0-py2.py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/pbr-5.6.0-py2.py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/pep8_naming-0.12.0-py2.py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/pluggy-0.13.1-py2.py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/poetry_semver-0.1.0-py2.py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/ptyprocess-0.7.0-py2.py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/py-1.10.0-py2.py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/pycodestyle-2.7.0-py2.py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/pydocstyle-6.1.1-py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/pyflakes-2.3.1-py2.py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/Pygments-2.9.0-py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/pymdown_extensions-8.2-py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/pyparsing-2.4.7-py2.py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/pytest-6.2.4-py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/pytest_cov-2.12.1-py2.py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/pytest_forked-1.3.0-py2.py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/pytest_randomly-3.8.0-py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/pytest_sugar-0.9.4-py2.py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/pytest_xdist-2.3.0-py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/python_dateutil-2.8.2-py2.py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/pytkdocs-0.11.1-py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/PyYAML-5.4.1-cp39-cp39-manylinux1_x86_64/lib
/home/pawamoy/.cache/pdm/packages/pyyaml_env_tag-0.1-py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/regex-2021.8.3-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64/lib
/home/pawamoy/.cache/pdm/packages/requests-2.26.0-py2.py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/safety-1.10.3-py2.py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/setuptools-57.4.0-py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/six-1.16.0-py2.py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/skjold-0.3.2-py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/smmap-4.0.0-py2.py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/snowballstemmer-2.1.0-py2.py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/stevedore-3.3.0-py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/termcolor-1.1.0-py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/toml-0.10.2-py2.py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/typed_ast-1.4.3-cp39-cp39-manylinux1_x86_64/lib
/home/pawamoy/.cache/pdm/packages/typing_extensions-3.10.0.0-py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/urllib3-1.26.6-py2.py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/watchdog-2.1.3-py3-none-manylinux2014_x86_64/lib
/home/pawamoy/.cache/pdm/packages/wps_light-0.15.2-py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/zipp-3.5.0-py3-none-any/lib
/home/pawamoy/.local/lib/python3.9/site-packages
/home/pawamoy/.basher-packages/pyenv/pyenv/versions/3.9.5/lib/python3.9/site-packages

Now, since I run VSCode using the same system user, and since VSCode seems to start Python process in my user's default shell (currently Zsh), all this machinery is enabled in VSCode as well. Indeed, if I select my system Python as the interpreter, and I run the following file using the following run configuration:

import sys

print("\n".join(sys.path))
        {
            "name": "Python: Current File",
            "type": "python",
            "request": "launch",
            "program": "${file}",
            "console": "integratedTerminal"
        }

It prints the same paths, except for some of them which are now the system paths instead of the pyenv ones:

%  cd /media/data/dev/python/duty ; /usr/bin/env /usr/bin/python /home/pawamoy/.vscode/extensions/ms-python.python-2021.8.1147840270/pythonFiles/lib/python/debugpy/launcher 43833 -- /media/data/dev/python/duty/pathfiles.py 
/media/data/dev/python/duty
/usr/lib/python39.zip
/usr/lib/python3.9
/usr/lib/python3.9/lib-dynload
/media/data/dev/python/duty/__pypackages__/3.9/lib
/home/pawamoy/.cache/pdm/packages/ansimarkup-1.5.0-py2.py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/appdirs-1.4.4-py2.py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/astor-0.8.1-py2.py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/attrs-21.2.0-py2.py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/autoflake-1.4-py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/bandit-1.7.0-py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/black-20.8b1-py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/certifi-2021.5.30-py2.py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/charset_normalizer-2.0.4-py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/click-8.0.1-py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/colorama-0.4.4-py2.py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/coverage-5.5-cp39-cp39-manylinux2010_x86_64/lib
/home/pawamoy/.cache/pdm/packages/darglint-1.8.0-py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/dparse-0.5.1-py3-none-any/lib
/media/data/dev/python/duty/src
/home/pawamoy/.cache/pdm/packages/execnet-1.9.0-py2.py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/failprint-0.8.0-py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/flake8-3.9.2-py2.py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/flake8_bandit-2.1.2-py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/flake8_black-0.2.3-py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/flake8_bugbear-21.4.3-py36.py37.py38-none-any/lib
/home/pawamoy/.cache/pdm/packages/flake8_builtins-1.5.3-py2.py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/flake8_comprehensions-3.5.0-py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/flake8_docstrings-1.6.0-py2.py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/flake8_plugin_utils-1.3.2-py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/flake8_polyfill-1.0.2-py2.py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/flake8_pytest_style-1.5.0-py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/flake8_string_format-0.3.0-py2.py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/flake8_tidy_imports-4.3.0-py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/flake8_variables_names-0.0.4-py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/ghp_import-2.0.1-py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/git_changelog-0.4.2-py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/gitdb-4.0.7-py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/GitPython-3.1.20-py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/idna-3.2-py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/importlib_metadata-4.6.3-py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/iniconfig-1.1.1-py2.py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/isort-5.8.0-py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/Jinja2-2.11.3-py2.py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/Markdown-3.3.4-py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/MarkupSafe-2.0.1-cp39-cp39-manylinux2010_x86_64/lib
/home/pawamoy/.cache/pdm/packages/mccabe-0.6.1-py2.py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/mergedeep-1.3.4-py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/mkdocs-1.2.2-py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/mkdocs_autorefs-0.2.1-py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/mkdocs_coverage-0.2.4-py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/mkdocs_gen_files-0.3.3-py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/mkdocs_literate_nav-0.4.0-py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/mkdocs_material-7.2.2-py2.py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/mkdocs_material_extensions-1.0.1-py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/mkdocstrings-0.15.2-py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/mypy-0.910-cp39-cp39-manylinux2010_x86_64/lib
/home/pawamoy/.cache/pdm/packages/mypy_extensions-0.4.3-py2.py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/packaging-21.0-py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/pathspec-0.9.0-py2.py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/pbr-5.6.0-py2.py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/pep8_naming-0.12.0-py2.py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/pluggy-0.13.1-py2.py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/poetry_semver-0.1.0-py2.py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/ptyprocess-0.7.0-py2.py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/py-1.10.0-py2.py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/pycodestyle-2.7.0-py2.py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/pydocstyle-6.1.1-py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/pyflakes-2.3.1-py2.py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/Pygments-2.9.0-py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/pymdown_extensions-8.2-py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/pyparsing-2.4.7-py2.py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/pytest-6.2.4-py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/pytest_cov-2.12.1-py2.py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/pytest_forked-1.3.0-py2.py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/pytest_randomly-3.8.0-py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/pytest_sugar-0.9.4-py2.py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/pytest_xdist-2.3.0-py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/python_dateutil-2.8.2-py2.py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/pytkdocs-0.11.1-py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/PyYAML-5.4.1-cp39-cp39-manylinux1_x86_64/lib
/home/pawamoy/.cache/pdm/packages/pyyaml_env_tag-0.1-py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/regex-2021.8.3-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64/lib
/home/pawamoy/.cache/pdm/packages/requests-2.26.0-py2.py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/safety-1.10.3-py2.py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/setuptools-57.4.0-py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/six-1.16.0-py2.py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/skjold-0.3.2-py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/smmap-4.0.0-py2.py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/snowballstemmer-2.1.0-py2.py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/stevedore-3.3.0-py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/termcolor-1.1.0-py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/toml-0.10.2-py2.py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/typed_ast-1.4.3-cp39-cp39-manylinux1_x86_64/lib
/home/pawamoy/.cache/pdm/packages/typing_extensions-3.10.0.0-py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/urllib3-1.26.6-py2.py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/watchdog-2.1.3-py3-none-manylinux2014_x86_64/lib
/home/pawamoy/.cache/pdm/packages/wps_light-0.15.2-py3-none-any/lib
/home/pawamoy/.cache/pdm/packages/zipp-3.5.0-py3-none-any/lib
/home/pawamoy/.local/lib/python3.9/site-packages
/usr/lib/python3.9/site-packages

In the end, if VSCode queries Python to get the paths to scan, shouldn't extra paths work for autocomplete and analysis as well, even with .pth files in them?

vscode
As you can see, VSCode cannot find the failprint package, which is nevertheless in sys.path (/home/pawamoy/.cache/pdm/packages/failprint-0.8.0-py3-none-any/lib).

But maybe I'm missing something here, like different execution environments/conditions?

@jakebailey
Copy link
Member

Can you provide trace logs from Pylance so we can see what interpreter and paths it is using?

"python.analysis.logLevel": "Trace"

@pawamoy
Copy link
Author

pawamoy commented Aug 23, 2021

Something like this?

[Info  - 8:28:17 PM] Pylance language server 2021.8.2 (pyright f462f5bb) starting
[Info  - 8:28:17 PM] Server root directory: /home/pawamoy/.vscode/extensions/ms-python.vscode-pylance-2021.8.2/dist
[Info  - 8:28:17 PM] No configuration file found.
[Info  - 8:28:17 PM] pyproject.toml file found at /media/data/dev/python/duty.
[Info  - 8:28:17 PM] Setting pythonPath for service "duty": "/usr/bin/python"
[Info  - 8:28:17 PM] Loading pyproject.toml file at /media/data/dev/python/duty/pyproject.toml
[Error - 8:28:17 PM] Pyproject file "/media/data/dev/python/duty/pyproject.toml" is missing "[tool.pyright] section.
[Warn  - 8:28:17 PM] stubPath /media/data/dev/python/duty/typings is not a valid directory.
[Info  - 8:28:17 PM] Assuming Python version 3.9
[Info  - 8:28:17 PM] Assuming Python platform Linux
Search paths for /media/data/dev/python/duty
  /home/pawamoy/.vscode/extensions/ms-python.vscode-pylance-2021.8.2/dist/typeshed-fallback/stdlib
  /media/data/dev/python/duty
  /media/data/dev/python/duty/src
  /media/data/dev/python/duty/__pypackages__/3.9/lib
  /home/pawamoy/.local/pipx/venvs/pdm/lib/python3.9/site-packages/pdm/pep582
  /media/data/dev/python/duty/typings
  /home/pawamoy/.vscode/extensions/ms-python.vscode-pylance-2021.8.2/dist/typeshed-fallback/stubs/...
  /home/pawamoy/.vscode/extensions/ms-python.vscode-pylance-2021.8.2/dist/bundled/stubs
  /usr/lib/python3.9
  /usr/lib/python3.9/lib-dynload
  /home/pawamoy/.local/lib/python3.9/site-packages
  /usr/lib/python3.9/site-packages
[Warn  - 8:28:17 PM] Exception received when installing file system watcher: TypeError [ERR_FEATURE_UNAVAILABLE_ON_PLATFORM]: The feature watch recursively is unavailable on the current platform, which is being used to run Node.js
[Warn  - 8:28:17 PM] Exception received when installing file system watcher: TypeError [ERR_FEATURE_UNAVAILABLE_ON_PLATFORM]: The feature watch recursively is unavailable on the current platform, which is being used to run Node.js
[Warn  - 8:28:17 PM] Exception received when installing file system watcher: TypeError [ERR_FEATURE_UNAVAILABLE_ON_PLATFORM]: The feature watch recursively is unavailable on the current platform, which is being used to run Node.js
[Warn  - 8:28:17 PM] Exception received when installing file system watcher: TypeError [ERR_FEATURE_UNAVAILABLE_ON_PLATFORM]: The feature watch recursively is unavailable on the current platform, which is being used to run Node.js
[Info  - 8:28:17 PM] Searching for source files
[Info  - 8:28:17 PM] Found 37 source files
[Info  - 8:28:17 PM] Background analysis(1) root directory: /home/pawamoy/.vscode/extensions/ms-python.vscode-pylance-2021.8.2/dist
[Info  - 8:28:17 PM] Background analysis(1) started
Background analysis message: setConfigOptions
Background analysis message: setImportResolver
Background analysis message: ensurePartialStubPackages
Background analysis message: setTrackedFiles
Background analysis message: markAllFilesDirty
Background analysis message: setFileOpened
Background analysis message: getSemanticTokens full
[BG(1)] getSemanticTokens full at /media/data/dev/python/duty/pathfiles.py ...
[BG(1)]   parsing: /media/data/dev/python/duty/pathfiles.py (22ms)
[FG] parsing: /media/data/dev/python/duty/pathfiles.py (18ms)
[FG] parsing: /home/pawamoy/.vscode/extensions/ms-python.vscode-pylance-2021.8.2/dist/typeshed-fallback/stdlib/builtins.pyi [fs read 2ms] (67ms)
[FG] binding: /home/pawamoy/.vscode/extensions/ms-python.vscode-pylance-2021.8.2/dist/typeshed-fallback/stdlib/builtins.pyi (29ms)
[FG] binding: /media/data/dev/python/duty/pathfiles.py (0ms)
[BG(1)]   parsing: /home/pawamoy/.vscode/extensions/ms-python.vscode-pylance-2021.8.2/dist/typeshed-fallback/stdlib/builtins.pyi [fs read 2ms] (71ms)
[BG(1)]   binding: /home/pawamoy/.vscode/extensions/ms-python.vscode-pylance-2021.8.2/dist/typeshed-fallback/stdlib/builtins.pyi (29ms)
[BG(1)]   binding: /media/data/dev/python/duty/pathfiles.py (0ms)
[BG(1)]   parsing: /home/pawamoy/.vscode/extensions/ms-python.vscode-pylance-2021.8.2/dist/typeshed-fallback/stdlib/typing.pyi [fs read 2ms] (27ms)
[BG(1)]   binding: /home/pawamoy/.vscode/extensions/ms-python.vscode-pylance-2021.8.2/dist/typeshed-fallback/stdlib/typing.pyi (8ms)
[BG(1)]   parsing: /home/pawamoy/.vscode/extensions/ms-python.vscode-pylance-2021.8.2/dist/typeshed-fallback/stdlib/_typeshed/__init__.pyi [fs read 0ms] (6ms)
[BG(1)]   binding: /home/pawamoy/.vscode/extensions/ms-python.vscode-pylance-2021.8.2/dist/typeshed-fallback/stdlib/_typeshed/__init__.pyi (1ms)
[BG(1)]   parsing: /home/pawamoy/.vscode/extensions/ms-python.vscode-pylance-2021.8.2/dist/typeshed-fallback/stdlib/typing_extensions.pyi [fs read 0ms] (3ms)
[BG(1)]   binding: /home/pawamoy/.vscode/extensions/ms-python.vscode-pylance-2021.8.2/dist/typeshed-fallback/stdlib/typing_extensions.pyi (1ms)
[BG(1)]   parsing: /home/pawamoy/.vscode/extensions/ms-python.vscode-pylance-2021.8.2/dist/typeshed-fallback/stdlib/sys.pyi [fs read 0ms] (6ms)
[BG(1)]   binding: /home/pawamoy/.vscode/extensions/ms-python.vscode-pylance-2021.8.2/dist/typeshed-fallback/stdlib/sys.pyi (4ms)
[BG(1)] getSemanticTokens full at /media/data/dev/python/duty/pathfiles.py (204ms)
Background analysis message: getSemanticTokens range
[BG(1)] getSemanticTokens range 0:0 - 3:0 at /media/data/dev/python/duty/pathfiles.py (0ms)
Background analysis message: analyze
[BG(1)] analyzing: /media/data/dev/python/duty/pathfiles.py ...
[BG(1)]   checking: /media/data/dev/python/duty/pathfiles.py (1ms)
[BG(1)] analyzing: /media/data/dev/python/duty/pathfiles.py (2ms)
Background analysis message: resumeAnalysis
Background analysis message: setFileOpened
Background analysis message: getDiagnosticsForRange
Background analysis message: getDiagnosticsForRange
[FG] parsing: /media/data/dev/python/duty/src/duty/cli.py (22ms)
[FG] binding: /media/data/dev/python/duty/src/duty/cli.py (3ms)
Background analysis message: getSemanticTokens full
[BG(1)] getSemanticTokens full at /media/data/dev/python/duty/src/duty/cli.py ...
[BG(1)]   parsing: /media/data/dev/python/duty/src/duty/cli.py (34ms)
[BG(1)]   binding: /media/data/dev/python/duty/src/duty/cli.py (3ms)
[BG(1)]   parsing: /home/pawamoy/.vscode/extensions/ms-python.vscode-pylance-2021.8.2/dist/typeshed-fallback/stdlib/argparse.pyi [fs read 0ms] (69ms)
[BG(1)]   binding: /home/pawamoy/.vscode/extensions/ms-python.vscode-pylance-2021.8.2/dist/typeshed-fallback/stdlib/argparse.pyi (5ms)
[BG(1)]   parsing: /home/pawamoy/.vscode/extensions/ms-python.vscode-pylance-2021.8.2/dist/typeshed-fallback/stdlib/inspect.pyi [fs read 0ms] (11ms)
[BG(1)]   binding: /home/pawamoy/.vscode/extensions/ms-python.vscode-pylance-2021.8.2/dist/typeshed-fallback/stdlib/inspect.pyi (5ms)
[BG(1)]   parsing: /home/pawamoy/.vscode/extensions/ms-python.vscode-pylance-2021.8.2/dist/typeshed-fallback/stdlib/textwrap.pyi [fs read 0ms] (3ms)
[BG(1)]   binding: /home/pawamoy/.vscode/extensions/ms-python.vscode-pylance-2021.8.2/dist/typeshed-fallback/stdlib/textwrap.pyi (1ms)
[BG(1)]   parsing: /media/data/dev/python/duty/src/duty/collection.py [fs read 0ms] (11ms)
[BG(1)]   binding: /media/data/dev/python/duty/src/duty/collection.py (2ms)
[BG(1)]   parsing: /media/data/dev/python/duty/src/duty/exceptions.py [fs read 0ms] (0ms)
[BG(1)]   binding: /media/data/dev/python/duty/src/duty/exceptions.py (0ms)
[BG(1)]   parsing: /media/data/dev/python/duty/src/duty/validation.py [fs read 0ms] (9ms)
[BG(1)]   binding: /media/data/dev/python/duty/src/duty/validation.py (1ms)
[BG(1)]   parsing: /home/pawamoy/.vscode/extensions/ms-python.vscode-pylance-2021.8.2/dist/typeshed-fallback/stdlib/abc.pyi [fs read 0ms] (1ms)
[BG(1)]   binding: /home/pawamoy/.vscode/extensions/ms-python.vscode-pylance-2021.8.2/dist/typeshed-fallback/stdlib/abc.pyi (0ms)
[BG(1)] getSemanticTokens full at /media/data/dev/python/duty/src/duty/cli.py (211ms)
Background analysis message: getSemanticTokens range
[BG(1)] getSemanticTokens range 0:0 - 48:28 at /media/data/dev/python/duty/src/duty/cli.py (3ms)
Background analysis message: getDiagnosticsForRange
Background analysis message: getDiagnosticsForRange
Background analysis message: analyze
[BG(1)] analyzing: /media/data/dev/python/duty/src/duty/cli.py ...
[BG(1)]   checking: /media/data/dev/python/duty/src/duty/cli.py (10ms)
[BG(1)] analyzing: /media/data/dev/python/duty/src/duty/cli.py (10ms)
Background analysis message: resumeAnalysis
Background analysis message: getDiagnosticsForRange
Background analysis message: getDiagnosticsForRange
[FG] parsing: /home/pawamoy/.vscode/extensions/ms-python.vscode-pylance-2021.8.2/dist/typeshed-fallback/stdlib/typing.pyi [fs read 0ms] (53ms)
[FG] binding: /home/pawamoy/.vscode/extensions/ms-python.vscode-pylance-2021.8.2/dist/typeshed-fallback/stdlib/typing.pyi (9ms)
[FG] parsing: /home/pawamoy/.vscode/extensions/ms-python.vscode-pylance-2021.8.2/dist/typeshed-fallback/stdlib/_typeshed/__init__.pyi [fs read 0ms] (8ms)
[FG] binding: /home/pawamoy/.vscode/extensions/ms-python.vscode-pylance-2021.8.2/dist/typeshed-fallback/stdlib/_typeshed/__init__.pyi (1ms)
[FG] parsing: /home/pawamoy/.vscode/extensions/ms-python.vscode-pylance-2021.8.2/dist/typeshed-fallback/stdlib/typing_extensions.pyi [fs read 0ms] (4ms)
[FG] binding: /home/pawamoy/.vscode/extensions/ms-python.vscode-pylance-2021.8.2/dist/typeshed-fallback/stdlib/typing_extensions.pyi (1ms)
[FG] parsing: /home/pawamoy/.vscode/extensions/ms-python.vscode-pylance-2021.8.2/dist/typeshed-fallback/stdlib/inspect.pyi [fs read 0ms] (10ms)
[FG] binding: /home/pawamoy/.vscode/extensions/ms-python.vscode-pylance-2021.8.2/dist/typeshed-fallback/stdlib/inspect.pyi (3ms)
Background analysis message: getDiagnosticsForRange
Background analysis message: getDiagnosticsForRange
[FG] parsing: /home/pawamoy/.vscode/extensions/ms-python.vscode-pylance-2021.8.2/dist/typeshed-fallback/stdlib/textwrap.pyi [fs read 0ms] (3ms)
[FG] binding: /home/pawamoy/.vscode/extensions/ms-python.vscode-pylance-2021.8.2/dist/typeshed-fallback/stdlib/textwrap.pyi (2ms)
Background analysis message: getDiagnosticsForRange
Background analysis message: getDiagnosticsForRange
Background analysis message: markFilesDirty
Background analysis message: analyze

@jakebailey
Copy link
Member

Yes, exactly. Is /usr/bin/python the same python that you're accessing in the terminal? Most of those paths are just from sys.path from calling python, so I would think that we get all of those paths that your one-liners are printing.

This could also be an instance of your python being special and handling __pypackages__; we don't actually run python within the workspace, we run it in a non-user folder as an extra security measure to prevent random code execution (in addition to other measures).

@pawamoy
Copy link
Author

pawamoy commented Aug 23, 2021 •
edited

In a terminal python points to the pyenv-installed one, but the results are the same if I just use /usr/bin/python instead.

we don't actually run python within the workspace, we run it in a non-user folder as an extra security measure to prevent random code execution (in addition to other measures).

That could explain the difference, but then again the Pylance logs show that /home/pawamoy/.local/pipx/venvs/pdm/lib/python3.9/site-packages/pdm/pep582 is in sys.path. If it does not come from my user's shell configuration (exported PYTHONPATH environment variable), where does it come from? And why paths under PDM's cache folder are not added to sys.path? They should be thanks to the sitecustomize.py in /home/pawamoy/.local/pipx/venvs/pdm/lib/python3.9/site-packages/pdm/pep582 . I'm confused 😕

EDIT: ah! wait. If you don't run it in the workspace, then the sitecustomize.py cannot find the __pypackages__ directory indeed. That answers what I just asked. And then .pth files are not supported with extraPaths. That explains everything. The question is now: is this a use-case you would like to support, and how should it be done?

@jakebailey
Copy link
Member

The fact that this whole thing relies on the directory you execute python in is really problematic (even further than what I mentioned in #1441, as now we can't even trust the interpreter to do the right thing).

I'm not entirely certain what to do here. We could try and remove our code that prevents the running of the python interpreter within the workspace and rely on other code execution protection measures, so that we do run python at the workspace root. That would then allow this custom stuff to find __pypackages__, but overall that doesn't fix __pypackages__ as a concept, because that PEP allows for much more magical sys.path changes based on where you are and what script you're running, which are very very hard to handle in an editor (where we just say "well, it's the workspace root"). Maybe it's not going to be common that a workspace contains more than one __pypackages__, and discovering based on the workspace root is enough, I don't know.

We could reuse the code we use for venvPath to parse out non-code-executing .pth files (just paths), with the assumption that users will be adding __pypackages__ to extraPaths (like you did), which would work for now, but isn't a long-term solution if __pypackages__ gets any traction. We don't want people having to configure paths just to match what the interpreter is supposed to be doing.

@pawamoy
Copy link
Author

pawamoy commented Aug 23, 2021 •
edited

I see. Getting the expected paths in a workspace is probably much more complicated than I can imagine. I would be OK with the second solution you mention. PDM itself documents that you need to add extra path to work in VSCode. For a better, long-term solution, I guess we'd have to engage with core Python developers, at https://discuss.python.org/t/pep-582-python-local-packages-directory/963 (if you are not already doing so ☺️), as well as with @frostming regarding PDM specifically.

Thank you very much @jakebailey for your time and help! Let me know if I can help in any way if you wish to proceed forward.

@jakebailey
Copy link
Member

I've implemented a prototype of the second solution and it does work (for simple pth files); we'll discuss whether or not it's a good idea or if the first solution is more robust.

@pawamoy
Copy link
Author

pawamoy commented Aug 24, 2021

I'll gladly test your prototype if it's possible! Just let me know 🙂 And thanks!

@jakebailey
Copy link
Member

We're probably going to take that second solution for now, as it's the least destructive.

One consideration is handling recursive pths, meaning pths that point to directories that also contain pth files.

I haven't been able to get PDM set up on my machine yet; can you see if you install matplotlib what your search paths come out to be? I believe they include a pth file in their regular release.

@frostming
Copy link

frostming commented Aug 25, 2021 •
edited

One consideration is handling recursive pths, meaning pths that point to directories that also contain pth files.

@jakebailey That is not necessary IMHO. The Python interpreter recognizes all .pth files under a path by site.add_site_dir(). It only adds the paths contained in a pth file to the end of sys.path and won't look into them recursively.

Thanks for your effort again.

@jakebailey
Copy link
Member

Great, I'll get my fix in for tomorrow.

I do wonder how that affects libs that do contain pth files; would PDM see this and fall back on regular copying? This is somewhat similar to how yarn works with PnP.

@frostming
Copy link

I do wonder how that affects libs that do contain pth files; would PDM see this and fall back on regular copying? This is somewhat similar to how yarn works with PnP.

Yes, .pth files are copied

@jakebailey jakebailey added the fixed in next version (main) A fix has been implemented and will appear in an upcoming version label Aug 25, 2021
@jakebailey
Copy link
Member

The next release will parse out pth files in extraPath folders into extraPaths as well, so this should be fixed.

Further discussion of PEP 582 in general can go on #1441.

@pawamoy
Copy link
Author

pawamoy commented Aug 25, 2021

Thanks a lot @jakebailey, that was fast!

@heejaechang
Copy link
Contributor

the fix is in 2021.8.3 release.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request fixed in next version (main) A fix has been implemented and will appear in an upcoming version
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@heejaechang @karthiknadig @pawamoy @jakebailey @frostming