Find library stubs in __pypackages__

[pawamoy]

Feature

It seems mypy cannot find library stubs when developing a project using PEP582, for example through PDM. Not opening as a bug but rather as a feature request because PEP582 is still quite recent and therefore expected not to be supported everywhere.

$ pdm add -ds typing types-Jinja2
Adding packages to typing dev-dependencies: types-jinja2
✔ 🔒 Lock successful
Changes are written to pdm.lock.
Changes are written to pyproject.toml.
Synchronizing working set with lock file: 2 to add, 0 to update, 0 to remove

  ✔ Install types-Jinja2 2.11.0 successful
  ✔ Install types-MarkupSafe 1.1.0 successful

🎉 All complete!

$ ls __pypackages__/3.6/lib/jinja2-stubs/
bccache.pyi  compiler.pyi   debug.pyi     environment.pyi  ext.pyi      __init__.pyi  loaders.pyi    meta.pyi   optimizer.pyi  runtime.pyi  _stringdefs.pyi  utils.pyi
_compat.pyi  constants.pyi  defaults.pyi  exceptions.pyi   filters.pyi  lexer.pyi     METADATA.toml  nodes.pyi  parser.pyi     sandbox.pyi  tests.pyi        visitor.pyi

$ pdm run  mypy --config-file config/mypy.ini src/project tests duties.py
src/project/redacted.py:8: error: Library stubs not installed for "jinja2.sandbox" (or incompatible with Python 3.6)
src/project/redacted.py:8: note: Hint: "python3 -m pip install types-Jinja2"
src/project/redacted.py:8: note: (or run "mypy --install-types" to install all missing stub packages)
src/project/redacted.py:8: note: See https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-imports
Found 1 error in 1 file (checked 16 source files)

Pitch

Support running mypy in projects managed by tools like PDM that take advantage of PEP582.

[pawamoy]

I've tried setting MYPY_PATH to __pypackages__/3.6/lib but of course it did not work, as expected and documented.
Then I tried creating a temporary directory and symlinking only the -stubs packages inside that temp dir, and set MYPY_PATH to that temp dir, but again, Mypy does not acknowledge the stubs 😕
If anyone has another workaround idea, please share!

[pawamoy]

Is there a way to ignore such errors in the mean time?

UPDATE: the following configuration seems to do the trick.

[mypy-jinja2.*]
ignore_missing_imports = True

[blueyed]

The variable to set is MYPYPATH (without underscore).

But it still does not work:

% MYPYPATH=$PWD/__pypackages__/3.9/lib mypy proj
mypy: "__pypackages__/3.9/lib/typing_extensions.py" shadows library module "typing_extensions"
note: A user-defined top-level module with name "typing_extensions" is not supported

It appears to work then creating a separate directory and linking django-stubs and django into it, and then use that with MYPYPATH:

% ls -l MYPYPATH
… django -> ../__pypackages__/3.9/lib/django/
… django-stubs -> ../__pypackages__/3.9/lib/django-stubs/
% MYPYPATH=$PWD/MYPYPATH mypy proj

/cc @frostming

[simon-fothergill]

I would love it if mypy became PEP 582 compatible.

You can install things like mypy and black elsewhere (e.g. globally with pipx) so they are not in __pypackages__ and then set MYPYPATH to look in the __pypackages__ directory.

I'm not sure if this has other side effects, though. It certainly type checks everything unless you set it to skip following all imports, which seems equivalent to ignoring missing imports. But you can silence following imports to give added safety.

Please feel free to comment on this approach.

Best wishes!

[pawamoy]

Using @blueyed solution, I've been able to automate that temporary directory creation with symlinks to packages of interest. It's implemented as a duty (my task runner), but could easily be rewritten for other task runners.

UPDATE: also support .pth files (install_cache feature of PDM)

@duty
def check_types(ctx):
    # NOTE: the following code works around this issue:
    # https://github.com/python/mypy/issues/10633

    # compute packages directory path
    py = f"{sys.version_info.major}.{sys.version_info.minor}"
    pkgs_dir = Path("__pypackages__", py, "lib").resolve()

    # build the list of available packages
    packages = {}
    for package in pkgs_dir.glob("*"):
        if package.suffix not in {".dist-info", ".pth"} and package.name != "__pycache__":
            packages[package.name] = package

    # handle .pth files
    for pth in pkgs_dir.glob("*.pth"):
        with suppress(OSError):
            for package in Path(pth.read_text().splitlines()[0]).glob("*"):
                if package.suffix != ".dist-info":
                    packages[package.name] = package

    # create a temporary directory to assign to MYPYPATH
    with tempfile.TemporaryDirectory() as tmpdir:

        # symlink the stubs
        ignore = set()
        for stubs in (path for name, path in packages.items() if name.endswith("-stubs")):
            Path(tmpdir, stubs.name).symlink_to(stubs, target_is_directory=True)
            # try to symlink the corresponding package
            # see https://www.python.org/dev/peps/pep-0561/#stub-only-packages
            pkg_name = stubs.name.replace("-stubs", "")
            if pkg_name in packages:
                ignore.add(pkg_name)
                Path(tmpdir, pkg_name).symlink_to(packages[pkg_name], target_is_directory=True)

        # create temporary mypy config to ignore stubbed packages
        newconfig = Path("config", "mypy.ini").read_text()
        newconfig += "\n" + "\n\n".join(f"[mypy-{pkg}.*]\nignore_errors=true" for pkg in ignore)
        tmpconfig = Path(tmpdir, "mypy.ini")
        tmpconfig.write_text(newconfig)

        # set MYPYPATH and run mypy
        os.environ["MYPYPATH"] = tmpdir
        ctx.run(f"mypy --config-file {tmpconfig} {PY_SRC}", title="Type-checking", pty=PTY)

There's no --ignore-errors=module command line option so I have to create a temporary config file to ignore the stubbed packages, otherwise mypy shows me errors for those.

[JaviFuentes94]

Any updates on this? It would be really helpful

[phiresky]

Does anyone have a workaround for modules that ship their own types? I can't figure out how to get mypy to not either throw

error: Cannot find implementation or library stub for module named "torch"

or to throw tons of errors for the code in the torch package itself.

I've tried linking the torch package to a directory that I then set to MYPYPATH.

[iyume]

I'm working on PEP582 globally and confused about the following error message after adding mypy_path = $MYPY_CONFIG_FILE_DIR/__pypackages__/3.9/lib to mypy.ini.

mypy: "__pypackages__/3.9/lib/typing_extensions.py" shadows library module "typing_extensions"

maybe a side effects?

[martimors]

Facing this issue with PEP582 (pdm package manager).

[patrick91]

Do you have this package installed by any chance? https://pypi.org/project/types-typing-extensions/

[iyume]

Do you have this package installed by any chance? pypi.org/project/types-typing-extensions

Yep, it exists at stdlib and my __pypackages__ (maybe some deps using it).

it exists at .local/lib/python3.9 and my __pypackages__

[patrick91]

@iyume removing it should fix the error :)

[fjarri]

What if it is a sub-dependency, and it does not shadow anything? (pdm sync was run on a clean Python installation, so typing_extensions only exists in __pypackages__).

[ethanhs]

Looking at the PEP discussion on discourse, it seems that a) the actual structure of the __pypackages__ directory is still not decided, so it would be pre-mature to implement something that we might have to change down the road and b) the PEP is stalled until someone has time to champion it and do all of the design work left to do.

So I don't think we can or should implement PEP 582 support.

However, I think #11143 should add support for __pypackages__ anyway, since the directory is on sys.path. I still don't think we should claim to support this use case (yet) but it should work after that PR is merged.

[CharString]

[off-topic]

it would be immature

I think you mean pre-mature. Sorry to ping everyone for this, but I like atmosphere in the Python community. Wouldn't want someone to zone out because their idea was called immature.

[ethanhs]

I think you mean pre-mature

Absolutely, thank you for catching that, I have edited my comment.

[gitpushdashf]

Looking at the PEP discussion on discourse, it seems that a) the actual structure of the __pypackages__ directory is still not decided, so it would be pre-mature to implement something that we might have to change down the road and b) the PEP is stalled until someone has time to champion it and do all of the design work left to do.

So I don't think we can or should implement PEP 582 support.

However, I think #11143 should add support for __pypackages__ anyway, since the directory is on sys.path. I still don't think we should claim to support this use case (yet) but it should work after that PR is merged.

This was merged in. Is it in 0.960/0.961? If so, does anyone know if it works?

[ethanhs]

That PR was recently merged, it will likely be in 0.970.

[idletea]

This was merged in. Is it in 0.960/0.961? If so, does anyone know if it works?

I'm running mypy 0.970+dev.914297e9486b141c01b34593938fdf423d892cef and it seems the issue's solved for me.

[hauntsaninja]

0.971 allows mypy to look at sys.path.

I don't think it's worth implementing any further support for PEP 582. PEP 582 is woefully underspecified, discussion seems stalled (draft PEP means very little), and in my opinion it's not the best approach anyway.