Move typing out of the stdlib in Python 3.7?

[gvanrossum]

I'm wondering if we should remove typing from the stdlib. Now's the time to think about this, as the feature freeze for 3.7 is about 12 weeks away.

Cons:

• People have to depend on a PyPI package to use typing (but they do anyway for typing_extensions)
• It's a backward incompatibility for users of Python 3.5 and 3.6 (but the typing module was always provisional)

Pros:

• The typing module can evolve much faster outside the stdlib
• We could get rid of typing_extensions (and maybe even mypy_extensions)

If we don't do this I worry that we're entering a period where many new typesystem features end up in typing_extensions and users will be confused about which items are in typing and which in typing_extensions (not to mention mypy_extensions). Anything new to be added to typing (e.g. Const, Final, Literal, or changing ABCs to Protocols) would have to be added to typing_extensions instead, and users would be confused about which features exist in which module. Moving typing out of the stdlib can make things potentially simpler, at the cost of an extra pip install (but they'll need one anyway for mypy).

Thoughts?

[vlasovskikh]

I haven't made up my mind yet. Here are some thoughts.

Extra cons:

• typing in the stdlib ensures that types are a part of the language standard and people can rely on them
• Type hints tend to introduce new language features like postponed evaluation of annotations, __class_getitem__ and so on
• The amount of new typing features could be overwhelming, sticking to Python releases sets a more comfortable pace keeping new stuff in typing_extensions and treating it like a beta

Extra pros:

• Python 2.7 already requires pip install typing

Brainstoming:

• typing_extensions could become a full clone of typing + new features
  • Provided that typing_extensions is always backwards compatible with typing
• Make the typing from the stdlib pip install -U updatable somehow

[JukkaL]

I don't have a strong opinion yet. Here are some additional things to think about:

• How would we standardize new typing features? Would they only be discussed here or would we continue to use the PEP process?
• What would happen to the existing typing-related PEPs if we'd discontinue relying on the PEP process?
• If we'd discontinue relying on the PEP process, where would specifications of features live? Would they perhaps be merged with the documentation?

Also, it would be nice to update 3.5 and 3.6 documentation to mention that typing is no longer part of the stdlib in 3.7, and add links to the standalone project.

[gvanrossum]

More discussion on python-dev: https://mail.python.org/pipermail/python-dev/2017-November/150110.html (observes that NamedTuple should be moved out then).

And python-ideas: https://mail.python.org/pipermail/python-ideas/2017-November/047655.html

[ambv]

Whoa, this is not something I expected to wake up to this morning :D And by cross-posting to python-dev and python-ideas you sure made it hard for yourself to keep track of the entire discussion. I'll post here, as "the smallest audience available to me." I hope this is fine.

In general, I think this is a good idea for the evolution of typing and it gives us opportunity to double-down on usability. But there are a few things that worry me.

PEP 563

If typing is removed from the standard library, so is get_type_hints(). This kills the PEP as not having a reliable, built-in way to evaluate annotations back is loss of functionality.

First I thought we could move the core functionality of get_type_hints() to a function in types. But if typing is out of the stdlib then it wouldn't be able to use this function itself because it wouldn't be available pre-3.7. And even post-3.7, there's a possibility it would want to move faster. So the equivalent shipped with Python 3.7 would be an ugly duckling anyway.

Psychological significance

"Removing" typing feels like defeat, like a failure of the provisional package. We might try to argue that it's about development velocity etc. but the emotional reaction is what it is. We added a module to Python and now we're removing it.

Way out

I have an idea how this all might fit together but I will need more time today to form it well.

[JelleZijlstra]

I'm also a bit worried about the psychological effect. People will see "typing is no longer in the stdlib" and conclude "oh, Python gave up on static typing".

[ambv]

I think I came to a conclusion. I think moving typing to PyPI is a reasonable choice only if coupled with a usability overhaul. It should improve the users' situation, not make it worse. Having to pip install typing and track version numbers is making it worse so we need to offset this with features that move the situation forward. I am -1 on "just moving" to PyPI as is.

Details on how I'd handle this are in #496.

[ambv]

Since there's so little time until beta 1, let's consider our options. If your main concern is the dreadful ergonomy of typing_extensions vs. frozen typing for 18 months then the easiest thing to do is to just keep PEP 484 provisional for another release. That gives us plenty of time to solve it right for 3.8.

If you really hate that option (why?) then the alternative is to figure out how to bundle typing in 3.7 forward in "ensurepip" style. In other words, provide a version with 3.7.0 but enable upgrading to a newer version with pip install -U. Hell, since this is provisional, we could even make this work for 3.6.4.

I wholeheartedly agree that typing_extensions will be awful if we don't do something now.

[ethanhs]

I agree that keeping PEP 484 provisional makes a lot of sense. I just re-read the relevant parts of PEP 411, which I see as a guideline of provisional packages, and similarly provisional peps.

The PEP says "Some reasons for not graduating are... the package may prove to be unstable". Also that "Withdrawals are expected to be rare."

I think we are all in agreement that there are some changes we would like to make to the API.

I also want to make an argument for keeping it provisional for the users' sake.

• If we move it to PyPI, a lot of code will break. People will have to deal with managing typing as a dependency, which will be a non-inconsiderable deterrent for the adoption of typing.

• If we accept it, the typing/typing_extensions dance will happen. Imo typing_extensions breaks the idea of "One-- and preferably only one --obvious way to do it.", as there are now multiple imports for typing information. There is more overhead to maintain a static typing (no pun intended) package.

• If we keep it provisional, the status quo remains. It isn't ideal to keep it provisional for a third release, but I think for the sake of users and the maintainers, it makes a lot of sense.

[Michael0x2a]

One thing to note is that even if we move typing out of the standard library, I don't think it'll be possible to get rid of or change typing_extensions, at least, unless...

1. We accept that people who want to support Python 3.5.x and 3.6.x can't use the latest features in typing. This was the status quo before typing_extensions was a thing.
2. We rename the typing module to something like "typing2" or provide an alternate download with a different name and tell anybody who wants to support 3.5.x and 3.6.x they can't use the "typing" module at all (since the version of typing bundled with the earlier versions of 3.5.x are incompatible with the latest version of typing, afaik). I believe proposal 2 was discussed and rejected from the original typing_extensions thread for basically this reason.

That said, perhaps these aren't too big prices to pay? If we go with option 1, for example, we'd basically be assuming that people only care able supporting the latest version of Python and perhaps Python 2.7, which I suppose isn't too unreasonable... (?)

It does add another barrier for people who want to convert an existing project over to using types though.

[ilevkivskyi]

I think we should not do this. Here are some thoughts:

1. "Psychological effect" is important. There are many people who hesitate whether to use typing and will not do this if it is not in stdlib (some already said this on python-dev). You may say that these people are not so important, but I think they are. The more people use static types, the more useful they are. For example, if many enough users of django will use static types, then this will convince django maintainers to provide stubs or inline annotations (using PEP 561) etc.

2. What exactly is the problem we are trying to solve? The provisional status has two main components, absence of backward compatibility guaranties and features between minor releases. Backward compatibility IMO is not a problem. It is hard to imagine that we will need to make e.g. Literal backward incompatible at runtime (mypy semantics still can change in an incompatible way). Moreover, there were some complains about lack of backwards compatibility, so I think it is important.

3. From previous it looks like the only aspect of provisional status that we really need is the ability to add features between minor releases. I think the best strategy that appeared here and on python-dev is to make typing non-provisional (including all backwards compatibility guarantees) but somehow make it upgradeable with pip install -U.

[gvanrossum]

Yes, the most promising plan is to keep it with the distributions but make it upgradable. I'm not sure if we could then claim it's no longer provisional -- non-provisional also forbids new features in bugfix releases. But regardless, the upgradability will remove most pressure to sync up with the CPython release cycle, allowing us to iterate faster.

[gvanrossum]

I've pondered this some more and it looks like there will be too many issues if we move typing out of the stdlib. Even if we kept it in the distribution bundles (like pip) there would be too many ways for users to shoot themselves in the foot, and for distributions to unbundle it, and it would send the wrong signal to people who want to start using type annotations.

So after long deliberation I am closing this issue. Instead we should keep the typing module provisional for at least 3.7 (we'll see where we stand in two years).

[ilevkivskyi]

One more thing that may still be worth discussing is do we need to add a "limited provisional" (or similar) status to PEP 411 that would mean like provisional but with backwards compatibility? Then we can declare that typing moves from provisional to limited provisional in Python 3.7.

[ambv]

Either it's stable or it's not. A state in-between seems both too limiting for fast iteration and yet still too volatile for conservative users to accept. As long as a package is provisional, it's not possible to accurately predict whether we'll need to break compatibility in the future.

[ilevkivskyi]

I would not call possibility to add new features in minor releases instability. This is the only thing we need. As I explained above it is highly unlikely that it will be necessary to make incompatible changes to the runtime typing API.

[gvanrossum]

I'm with Ivan. I don't think we need to update PEP 411, we can just put this in the docs for the typing module.

…

On Mon, Nov 13, 2017 at 9:13 AM, Ivan Levkivskyi ***@***.***> wrote: I would not call possibility to add new features in minor releases instability. This is the only thing we need. As I explained above it is highly unlikely that it will be necessary to make incompatible changes to the *runtime* typing API. — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub <#495 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACwrMpUj-orIzpXG5cW__DevJT-iFbyyks5s2HitgaJpZM4QRLKq> .

-- --Guido van Rossum (python.org/~guido)