ENH: Publish PEP 561 py.typed markers to allow downstream projects to use static code analysis

[Trolldemorted]

Is your feature request related to a problem? Please describe.

Downstream consumers cannot use static code analysis tools against their code that uses scipy.

Describe the solution you'd like.

If the type hints in your code are correct, all you have to do is include a PEP 561 py.typed marker, and your users will be able to type-check their usage of scipy types and functions.

Describe alternatives you've considered.

Publishing dedicated stub packages could also work, but compared to shipping marker files this is much more work

Additional context (e.g. screenshots, GIFs)

No response

[rgommers]

Thanks for opening this issue @Trolldemorted

If the type hints in your code are correct,

That is a very big IF. There are some type hints in the SciPy code base, but not many. See the huge list of exclusions in mypy.ini that are needed to make mypy scipy/ pass in CI.

It's actually not 100% clear to me from PEP 561 if it's possible to ship partial support. For example per SciPy submodule. This text implies it: "Package maintainers who wish to support type checking of their code MUST add a marker file named py.typed to their package supporting typing. This marker applies recursively: if a top-level package includes it, all its sub-packages MUST support type checking as well." However, it doesn't actually say that you may place a py.typed file in pkg/subpkg/ only. If you could figure out if that works, that'd be helpful.

[luator]

@rgommers I did a quick test and at least with mypy it seems to work. This is what I did:

Create the following package structure:

pkg
├── __init__.py
├── sub1
│   └── __init__.py
└── sub2
    ├── __init__.py
    └── py.typed

Content of pkg/sub1/__init__.py:

def foo1(x: int) -> float:
    return x / 1.5

Content of pkg/sub2/__init__.py:

def foo2(x: str) -> str:
    return x + x

The other files are empty.

Then, in a separate directory I created the following script:

from pkg.sub1 import foo1
from pkg.sub2 import foo2


f: float
s: str

f = foo1(42)  # good
s = foo2("hello")  # good
s = foo1("hello")  # bad
f = foo2(42)  # bad

Then run mypy on this script:

export PYTHONPATH=path/to/pkg
mypy script.py

which gives me the following output:

script.py:1: error: Skipping analyzing "pkg.sub1": module is installed, but missing library stubs or py.typed marker  [import]
script.py:1: note: See https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-imports
script.py:11: error: Incompatible types in assignment (expression has type "str", variable has type "float")  [assignment]
script.py:11: error: Argument 1 to "foo2" has incompatible type "int"; expected "str"  [arg-type]
Found 3 errors in 1 file (checked 1 source file)

So type hints in pkg.sub1 are ignored (and we correctly get an error regarding missing type info) while type hints of pkg.sub2 are used and we get the expected error for wrong use of foo2.

Tested with mypy 0.991.

[rgommers]

Great, thanks for that @luator! Okay, that means we can do this per submodule as soon as static typing support for a submodule is complete. The first two candidates are probably scipy.special (see gh-11749 for status) and scipy.spatial. The latter is complete enough to ship a py.typed file for today I think - all extension modules have stubs, and the ignore_errors exclusions in mypy.ini are only for test files.

[Avasam]

Downstream consumers cannot use static code analysis tools against their code that uses scipy

In the meantime, there's some stubs available at https://github.com/microsoft/python-type-stubs/tree/main/stubs/scipy-stubs that you can download and configure your type checker to point to the folder you installed them in.
Most will default to typings. In mypy you can set it explicitely using:

[mypy]
mypy_path = $MYPY_CONFIG_FILE_DIR/typings

There's an open feature request to publish those stubs on PyPI: microsoft/python-type-stubs#211

(As a note: as submodules typing are progressively completed and marked as such with py.typed, then published. They can be removed from https://github.com/microsoft/python-type-stubs/tree/main/scipy-stubs to provide a better experience to Pylance users)