Discuss: A module for experimental extensions to the "typing" module

[davidfstr]

This issue tracks a discussion originating in #2206 about creating a module for experimental extensions to the standard "typing" module.

[davidfstr]

In #2206 where I proposed the idea of a mypy.typing module to hold experimental extensions to the typing module, I had a few things in mind:

• Extending the typing module is difficult because I presume there is an expectation that anything in typing should be supported by all Python typecheckers. That implies that anything added to typing would be standardized in a PEP, such as PEP 484 (Type Hints), after community discussion.
• Nevertheless there is a desire to quickly try out some new typechecking extensions in the mypy typechecker without going through the full standardization process. Such extensions would presumably imply changes to the typing module or a different module that serves an analogous purpose.

Therefore I propose adding a mypy.typing module to hold mypy-specific extensions to typing. After an extension is added, an incubation period would follow whether the extensions were tried out by the mypy community to see whether they worked well. These extensions might be abandoned or they might later graduate to the standard typing module.

The preceding is one use case for extending the typing module.

---

@gvanrossum indicates that there is a preexisting --typing-module flag that allows a custom typing module (which presumably is a superset of the original one) to be swapped in for advanced users which would be available through the regular import typing statement. This feels like a slightly different use case, so I'd like to hear some elaboration on how the --typing-module flag is used currently and/or how it is intended to be used. It may reveal a different desired workflow for extending the typing module.

[gvanrossum]

Thanks for opening a new issue!

I'd also like to include by reference my comment from #2206, at least the part where I explained that the name of the new module should not be "mypy dot something" because of the meaning of the mypy package. I'm open for another name that has "mypy" in it, e.g. "mypy_extensions.py" -- I just don't think it should live in the mypy package.

We should also consider whether the new module should act as an extended copy of the typing module, or whether it only exports new extensions. IOW could we write "from mypy_extensions import Any, List, TypedDict" or would we have to write "from mypy import Any, List; from mypy_extensions import TypedDict". Each has some pros and cons, as you can imagine.

[davidfstr]

Indeed I note the comment that the mypy typechecker (and hence the mypy package itself) requires Python 3 yet it can be applied to typecheck code written with Python 2. That rules out the possibility of a mypy.typing module.

---

whether the new module should act as an extended copy of the typing module, or whether it only exports new extensions

I'd be in favor of a module that only has new extensions. Reasoning:

• A vendor.typing module with extensions only forces developers to import from the regular typing whenever using a standardized typing feature. If vendor.typing included all standardized features as well a developer might be tempted to import everything from it, obscuring which non-standardized features were being used (if any at all).

By analogy, imagine if a website used 100% vendored -webkit-somestyle CSS classes even when a non-vendored somestyle class would do. It would unnecessarily couple the website to the particular vendor implementation. I assert that use of vendored CSS extensions (and by analogy vendored typing extensions) should be discouraged when they are not necessary.

---

Another interesting question is whether it is only the typing module that needs to be vendored or whether there are others.

• If there are no other modules of interest and we are confident that there won't be in the future, we could have a top-level mypy_typing module.
• On the other hand if there are other modules currently or potentially of interest, we'd probably want them to be grouped into a top-level module like mypy_extensions.typing as mentioned earlier.

Since the future is uncertain, I'd lean toward the more future-proof mypy_extensions.typing arrangement.

---

One might quibble over the number of letters in mypy_extensions vs, for example, mypy_support or various other variations. I'm not particularly opinionated here.

[gvanrossum]

You've pretty much convinced me that the proposed module should only contain extensions. I've got some small worry left about a possible situation where an extension requires modification of an existing type (say, if we wanted to add some new flag to TypeVar). I guess in that case the extensions module should export a version that includes both the standard and the new functionality.

I'm also okay with naming the new thing mypy_extensions.typing. It should perhaps live in its own repo, preferably named python/mypy_extensions. Or it could be a subdirectory of the mypy repo with its own setup.py -- that's perhaps more convenient. (And perhaps mypy's own setup.py could also be modified to install it -- it does that for typing.py anyways when installed under 3.4 or lower.)

[davidfstr]

Or it could be a subdirectory of the mypy repo with its own setup.py -- that's perhaps more convenient.

Definitely more convenient. It would imply that mypy_extensions would be version-locked with mypy. I don't think this restriction is a problem because changes to mypy_extensions would be tightly coupled with mypy anyway.

Having mypy's setup.py install mypy_extensions automatically is an interesting idea I hadn't considered.

1. I assume that pip install mypy-lang doesn't work on Python 2. Therefore it would be necessary to install mypy_extensions extensions by itself so that Python 2 programs could use it. So pip install mypy_extensions would then need to work, implying that mypy_extensions would at least need to support separate distribution.
2. Given the above point, would pip3 install mypy-lang also automatically install mypy_extensions? I suppose it probably would since mypy would probably have some direct uses and therefore would list it as a dependency on PyPl.

[gvanrossum]

Currently "pip install mypy-lang" also installs types.py for the current Python 3 version, if that's < 3.5. So for Python 2 or if your Python 3 target is different than your the Python 3 version you use to run mypy, you still have to do "pip install types". We could make it so that "pip install mypy-lang" also installs mypy_extensions, but you could also run a separate "pip install mypy_extensions". All this could live in the mypy repo; the version-lock doesn't bother me (we should make sure that the separate install has a matching version).

[davidfstr]

Okay I like this approach. What would the directory structure look like? Maybe something like:

mypy/ (git root)
  mypy/ (package root)
    typing.py -- REMOVE
    ...
  mypy_extensions/ (vanilla directory) -- ADD
    mypy_extensions/ (package root)
      typing.py -- ADD
    setup.py
  typeshed/ (git submodule root)
  ...
  setup.py

This structure has the advantage of giving the mypy_extensions setup.py a nice home. But I suspect that the inability to import mypy_extensions.typing when at the top-level mypy directory could be annoying. I suspect there's at least one script depending on the ability to do python3 -m mypy ... at the top-level without changing the PYTHONPATH.

[gvanrossum]

• I'd like to avoid two levels of directories with the same name. It always confuses people or tools. Just pick a different name for the upper mypy_extensions.
• As long as mypy itself doesn't depend on its own extensions I think we should be fine. It's a classic bootstrapping problem. What other scripts would need to import it? The unittests can have their own stubs in test-data/unit/{lib-stubs,fixtures}.
• We also need stubs for this in typeshed, but that's not hard.

[davidfstr]

Just pick a different name for the upper mypy_extensions.

Maybe just "extensions" for the top-level?

The unittests can have their own stubs in test-data/unit/{lib-stubs,fixtures}.

Stubs? Do you mean mock objects of some kind?

We also need stubs for this in typeshed, but that's not hard.

I don't think I follow. If mypy_extensions/typing.py has inline type annotations, why would there need to be an external .pyi file?

[gvanrossum]

Ah, you haven't truly grokked the wonderful world of mypy unittests. For speed reasons they have very small custom stubs that are substituted for builtins.pyi by the test framework. That's why many tests have a [builtins ...] directive. Those files come from fixtures; the defaults live in lib-stubs. They are the source of many test conundrums but without them the tests would run much slower.

Regarding the need for a separate (real) typing.pyi, the reason is that file is full of hacks that mypy wouldn't like, so we don't type-check typing.py (same reason why mypy doesn't analyze the stdlib by default, and it's not a good idea to change that).

[JukkaL]

Cool, this could be very useful for other experimental features as well. Naming the top-level directory extensions sounds good to me.

By the way, PEP 8 discourages underscores in package names -- but is something like mypyextensions or mypyext even uglier?

[gvanrossum]

I would disregard PEP 8 here. That particular rule is probably really only valid for the stdlib.

[davidfstr]

Okay it sounds like we do need a typing.pyi. However could we put this typing.pyi adjacent to mypy_extensions/typing.py rather than having it in typeshed?

Typeshed feels like the place to put .pyis when the original library is not willing to provide its own annotations (either inline or via .pyi). In the case of mypy_extensions I think it would be easier for it to manage its own .pyi instead of externalizing it. Thoughts?

[JukkaL]

I think that it's okay to put it in typeshed, at least for now, because that's currently the only standard location from where other tools can find stubs. If a library has annotations, there's no good way to declare this fact, and users will likely have to manually configure their tool to look at the library implementation for annotations. (Of course mypy can special case the stubs for its internal stuff so that it will always find them.)