Document how to write test-cases for type annotations using mypy (in subprocess)

[sirosen]

Documentation

When writing a python package which publishes type information, it's very easy to mess up the annotations you present to your users. For example, a decorator can easily lose type information if written carelessly:

def mydeco(f: Callable) -> Callable:
    ...

The solution, "obviously", is to test your annotations. But how?
Although there are several packages which add pytest-mypy integrations, there is no guidance from mypy.

I propose the addition of a new doc on "testing package annotations" in the miscellaneous section, which demonstrates a simple test helper which writes provided code to files and runs mypy on those files, for use with test suites.

To seed things, here's what I wrote recently to do this:

import os
import re

import pytest

# mypy's only python-callable API
# see:
# https://mypy.readthedocs.io/en/stable/extending_mypy.html#integrating-mypy-into-another-python-application
from mypy import api as mypy_api


@pytest.fixture
def _in_tmp_path(tmp_path):
    cur = os.getcwd()
    try:
        os.chdir(tmp_path)
        yield
    finally:
        os.chdir(cur)


@pytest.fixture
def run_reveal_type(tmp_path, _in_tmp_path):
    content_path = tmp_path / "reveal_type_test.py"

    def func(code_snippet, *, preamble: str = ""):
        content_path.write_text(preamble + f"reveal_type({code_snippet})")
        result = mypy_api.run(["reveal_type_test.py"])
        stdout = result[0]
        # mypy reveal_type shows notes like
        #   note: Revealed type is "Union[builtins.str, None]"
        # find and extract with a regex
        match = re.search(r'note: Revealed type is "([^"]+)"', stdout)
        assert match is not None
        return match.group(1)

    return func

I'm leveraging pytest's fixture system here, but I'd be happy to write the necessary tmpdir bits to make this compatible with unittest or other frameworks.

I would be enthusiastic about documenting this and putting together a PR, provided that the maintainers here like the idea (and find the proposed testing helper to document acceptable).

[erictraut]

Thanks for the suggestion. Improving the quality of type information in packages benefits everyone, and investments in tools and documentation are welcome!

A few additional thoughts...

There has recently been a push to consolidate type-related documentation in the python/typing repo. The documentation that you're proposing might make more sense to post there. I guess it depends on how mypy-specific it is.

In addition to testing the correctness of type information, there's also the issue of "type completeness" — i.e. is a package missing type information for portions of its interface? If you're interested in validating both the correctness and completeness of your package's type annotations, I added a feature to pyright that you might be interested in checking out. For details, refer to this documentation.

[sirosen]

I wasn't aware of the shift towards docs on the typing repo. I think even though the current doc that I'd submit is mypy-specific, putting it there leaves room for the doc to expand to cover pyright, pyre, etc.

Do you think it makes sense to convert the doc on "type completeness" to a "testing type information" doc, and make completeness and inspection of specific types two separate sections?

[erictraut]

@srittau is the main person behind the recent improvements to the typing repo. I think he has a vision for the layout of the docs, so he'd be in the best position to answer your question.

From my perspective, I think it might be preferable to have one document that provides guidance to library authors about what is expected for typing within a "py.typed" package and a separate document that discusses tools, tests, automation, and other best practices. If others agree with that split, we could move the section about pyright's "--verifytypes" feature out of the existing document and into the new document that focuses on tools & tests.

[sobolevn]

As the main maintainer of https://github.com/typeddjango/pytest-mypy-plugins I am totally interested in helping with this doc 🙂

[srittau]

Just briefly (I haven't read through all comments here yet): I suggested a documentation structure in python/typing#845. @shannonzhu also provided an initial PR for a structure in python/typing#919 (which I also haven't found time to look at yet). I agree with Eric that we should generally split general typing documentation and type checker specific documentation. Two exception that come to mind: Tutorials should pick a type checker (most likely mypy) for experimentation and we could point out specific type checker quirks as footnotes or in callout boxes, if necessary.

[hauntsaninja]

Not quite what you're asking for, but mypy has a tool called stubtest that is pretty good at testing that type annotations match runtime objects. It's particularly useful if you're bundling pyi files, since those have a tendency to diverge.

If possible, running mypy with stricter settings on package code is also useful (e.g. --disallow-any-generics would complain about your example)

In general, what you're suggesting is very similar to how mypy's tests run. Could be great to try and factor out the relevant logic from there.

[sirosen]

Thanks everyone for thoughts and ideas! It sounds like this would come together as a page with a structure like

Tools for Working with Type Annotations
- Generating Annotations
  - stubgen
  - monkeytype? others?
- Testing Annotation Accuracy
  - reveal_type + subprocess
  - pytest-mypy-plugins
  - mypy stubtest
- Annotation Coverage
  - pyright --verifytypes
  - mypy --html-report / --txt-report

I can write a first draft sometime soon -- and I'll keep it all brief to make review and incremental improvements easier. But I'm fine with waiting for python/typing#919 to land before starting on this if that simplifies things.

[hauntsaninja]

Some prior art:
https://mypy.readthedocs.io/en/latest/stubgen.html
https://mypy.readthedocs.io/en/latest/installed_packages.html#creating-pep-561-compatible-packages

[ikonst]

Similar approaches I've seen / taken:

• tests that contain Python code snippets (rather than a separate YAML): https://github.com/pynamodb/PynamoDB/blob/master/tests/test_mypy.py
• tests that are syntactically Python code: https://github.com/davidfritzsche/pytest-mypy-testing (seemingly unmaintained)
• maintain separate typing_tests/ containing files which are simply processed by mypy, then configure mypy to run with warn_unused_ignores = true

The last case provides somewhat weaker guarantees, but I like the elegance of needing no additional tooling. For example, the following ensures that a decorator isn't a typing black hole:

@my_decorator()
def f(a: int) -> str:
    return str(a)

f.warm_up(42)
f.warm_up("42")  # type: ignore [arg-type]
f.warm_up(b=42)  # type: ignore [call-arg]
r1: str = f(42)
r2: int = f(42)  # type: ignore [assignment]

[sirosen]

python/typing#1071 just merged, so there's now a doc for this! 🎉