Add typing extensions #443

Michael0x2a · 2017-06-27T00:10:30Z

This pull request adds a typing_extensions subproject to typing. The typing_extensions module backports any new additions to typing for Python 3.5+ users who are using older versions of typing that were bundled with their standard library (and so can't update to the latest versions).

See #435 for more motivation and additional context.

ilevkivskyi · 2017-06-27T07:58:39Z

I don't think you need all these old versions bundled in tests. I think we should just properly configure Travis (it supports Python micro releases like 3.5.2 etc. (like in mypy's .travis.yml), and specific versions of typing to be installed before tests)

Michael0x2a · 2017-06-27T08:10:04Z

@ilevkivskyi I guess my only concern with that is that it becomes harder to do sanity checks locally -- having to constantly push to run tests seems a bit annoying to me.

ilevkivskyi · 2017-06-27T09:06:36Z

having to constantly push to run tests seems a bit annoying to me.

You can still keep these copies locally (without tracking them in git). But I would prefer a properly configured .travis.yml to test with actual Python versions (note that Python runtime changes itself between versions, not only the stdlib, there are few places in typing where this is actually important).

ilevkivskyi · 2017-06-27T09:58:55Z

typing_extensions/src_py3/typing_extensions.py

+T_contra = typing.TypeVar('T_contra', contravariant=True)  # Ditto contravariant.
+
+
+def _gorg(a):


Note that _gorg and _geqv will be removed soon after Python 3.6.2 is released, see PR #439.

Sounds good -- once this pull request gets accepted, I can prepare another pull request to add those changes and leave it pending just like #439 (or whatever other course of action you think is best).

or whatever other course of action you think is best

OK, let's just not forget about this.

ilevkivskyi · 2017-06-27T10:02:17Z

Concerning the test failure, we decided to stop supporting Python 3.2 (it reached EoL a year ago), so that you can just skip testing on Python 3.2.

Michael0x2a · 2017-06-27T10:04:28Z

Update: I removed run_tests.py and the associated machinery in favor of just using Travis, as suggested. The only remaining blocker is that we can't actually install typing on Python 3.2. I figured it'd be best to make a separate pull request for that though: #444

Once that's accepted, I can rebase, but this pull request should otherwise be ready for review (?).

Edit: lol, sniped

ilevkivskyi · 2017-06-27T10:14:03Z

I can review this, but I am not sure I will be able to review before week-end.

ilevkivskyi

Sorry for waiting with this, the PR looks very good. Thanks for work! I just have mostly minor comments.

In general I am eager to merge this soon, so that we can put stuff from mypy_extensions here as well. And Protocol+@runtime :-)

ilevkivskyi · 2017-07-07T13:41:44Z

typing_extensions/setup.py

+      author='Guido van Rossum, Jukka Lehtosalo, Lukasz Langa, Michael Lee',
+      author_email='jukka.lehtosalo@iki.fi',
+      # TODO: Change URL
+      url='https://github.com/michael0x2a/typing_extensions',


The link should be to the main typing repo.

Whoops, forgot to change that.

ilevkivskyi · 2017-07-07T13:42:42Z

typing_extensions/setup.py

+'''
+
+classifiers = [
+    'Development Status :: 5 - Production/Stable',


This is far from stable :-) I would put alpha here.

ilevkivskyi · 2017-07-07T13:45:01Z

tox.ini

+  src/test_typing.py,
+  typing_extensions/src_py2/test_typing_extensions.py,
+  typing_extensions/src_py3/test_typing_extensions.py,
+  # test_data should contain an exact snapshot of the standard


Do we still need this? I thought we agreed that we could leave this out and just use updated Travis config.

Same thing -- forgot to remove it.

ilevkivskyi · 2017-07-07T13:45:46Z

.travis.yml

@@ -14,5 +18,8 @@ install:

 script:
  - export PYTHONPATH=`python -c "import sys; print('python2' if sys.version.startswith('2') else 'src')"`; py.test $PYTHONPATH
+  - python setup.py install;


I am just curious, why this was not necessary before and is necessary now?

The problem is that the typing_extensions depends on typing being installed for Python 3.4 and below since we try and re-export as much as possible from the typing module. That said, I've tried adjusting travis so it should install typing only for Python 3.4 and below -- we'll see if I did it correctly once the tests are done.

ilevkivskyi · 2017-07-07T13:49:41Z

typing_extensions/README.md

+To run tests, navigate into the appropriate source directory and run
+`test_typing_extensions.py`. You will also need to install the latest
+version of `typing` if you are using a version of Python that does not
+include `typing` as a part of the standard library.


Maybe we could add a requirements.txt or test_requirements.txt for this?

I added it to the test_requirements.txt file for the typing repo. If you prefer, I can leave that one alone and add a separate test_requirements.txt and requirements.txt file to the typing_extensions folder.

Since typing_extensions is basically a separate project, maybe it makes sense for it to have its own requirement files, but it does feel a little redundant...

I would actually prefer separate requirements for typing_extensions.

ilevkivskyi · 2017-07-07T14:16:22Z

typing_extensions/src_py3/test_typing_extensions.py

+
+
+if __name__ == '__main__':
+   # main(argv=[sys.argv[0]] + sys.argv[2:])


Is this comment necessary? Why do we need this pattern?

This was a holdover from the code I previously had for running tests, which we removed. I removed the comment here + in the Python 2 tests.

ilevkivskyi · 2017-07-07T14:16:38Z

typing_extensions/src_py3/typing_extensions.py

+try:
+    import collections.abc as collections_abc
+except ImportError:
+    import collections as collections_abc  # Fallback for PY3.2.


Again, this is not needed.

Also removed.

ilevkivskyi · 2017-07-07T14:18:19Z

typing_extensions/src_py3/typing_extensions.py

+    _generic_new = typing._generic_new
+else:
+    def _generic_new(base_cls, cls, *args, **kwargs):
+        return base_cls.__new__(cls, *args, **kwargs)


I am not 100% sure this is a safe substitution. Do you have a test for this? IIRC, the point was that __init__ was not called at instantiation of subscripted generics.

I don't know if I necessarily have tests for specifically this -- I was mostly copying what the typing module was already doing and assumed the behavior was correct. For example, in Python 3.5.0 - 3.5.2, the List type was defined like this:

class List(list, MutableSequence[T], extra=list): def __new__(cls, *args, **kwds): if _geqv(cls, List): raise TypeError("Type List cannot be instantiated; " "use list() instead") return list.__new__(cls, *args, **kwds)

In Python 3.5.3+, it was defined like this:

class List(list, MutableSequence[T], extra=list): __slots__ = () def __new__(cls, *args, **kwds): if _geqv(cls, List): raise TypeError("Type List cannot be instantiated; " "use list() instead") return _generic_new(list, cls, *args, **kwds)

I'm also not sure if this substitution is 100% correct, but if it's broken, it'll at least be consistently broken in the same way the rest of the typing module was in Python 3.5.0 - 3.5.2. Defining the function this way also helped me reduce redundancy since I can define things like Deque just once instead of twice.

be consistently broken in the same way the rest of the typing module was in Python 3.5.0 - 3.5.2

OK, but then please mention this in a docstring/comment.

ilevkivskyi · 2017-07-07T14:19:42Z

typing_extensions/src_py3/typing_extensions.py

+    'NewType',
+    'overload',
+    'Text',
+    'TYPE_CHECKING',


Again, I am not sure about GenericMeta (see comment above).

ilevkivskyi · 2017-07-07T14:21:23Z

typing_extensions/src_py3/typing_extensions.py

+T_contra = typing.TypeVar('T_contra', contravariant=True)  # Ditto contravariant.
+
+
+def _gorg(a):


or whatever other course of action you think is best

OK, let's just not forget about this.

ilevkivskyi · 2017-07-07T14:31:03Z

Actually, I have one more suggestion. Maybe edit description to reflect the fact that experimental things that are going to appear in typing later will appear here first (like TypedDict, argspecs, and Protocol).

Michael0x2a · 2017-07-07T15:47:47Z

@ilevkivskyi Thanks for the review! I'll start going through your comments and making fixes in a moment.

gvanrossum · 2017-07-07T15:55:53Z

You will have to wait until after the release unless @JukkaL merges this despite the code freeze.

…

On Jul 7, 2017 8:47 AM, "Michael Lee" ***@***.***> wrote: @ilevkivskyi <https://github.com/ilevkivskyi> Thanks for the review! I'll start going through your comments and making fixes in a moment. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#443 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACwrMvzaBjfKPMxJS4jdNXcCf6T9eISxks5sLlMkgaJpZM4OF__7> .

JelleZijlstra · 2017-07-07T21:39:29Z

typing_extensions/src_py3/test_typing_extensions.py

-TYPING_V3 = sys.version_info >= (3, 5, 3)
-TYPING_V4 = sys.version_info >= (3, 6, 1)
+TYPING_LATEST = sys.version_info[:3] < (3, 5, 0)
+TYPING_V2 = TYPING_LATEST or sys.version_info[:3] >= (3, 5, 1)


I think this would be clearer as TYPING_3_5_1 or similar, because "v2" is not a terminology used elsewhere.

Michael0x2a · 2017-07-07T23:05:43Z

@ilevkivskyi -- Ok, I should be ready for a second round of review. I tried adding links to relevant pull requests and issues for the typing version-specific if statements, but found it quite hard to do so in general, so unfortunately most of them are still undocumented :(.

See latest comment in python/typing#443 for justification.

Michael0x2a · 2017-07-08T05:25:52Z

Just as an FYI, I decided to remove typing_extensions.Collection -- in retrospect, I think attempting to backport that type was a mistake.

In order to correctly backport that type, I basically need to inject typing_extensions.Collection into the middle of an existing inheritance hierarchy, which I don't think is possible to do cleanly, if at all. More concretely, the following code doesn't typecheck in Python 2 and older versions of Python 3 (the typing stubs have typing.Sequence subclass typing.Collection only some of the time):

from mypy_extensions import Collection

def foo(x):
    # type: Collection[int] -> int
    return sum(x)

foo([1, 2, 3])

I think this is potentially repairable at least at typechecking-time with some very careful surgery on the typing stubs, but doing that seems questionable to me. It also wouldn't fix any runtime isinstance or issubclass checks people might try doing.

(More specifically, Python 3's typing stubs has typing.Sequence always subclass a phantom _Collection type, which is only publicly exposed as typing.Collection for Python 3.6+. We could add a similar phantom type to the Python 2 typing stubs, then have the typing_extension re-export typing._Collection as typing_extensions.Collection. Users would then need to use an up-to-date version of typeshed.)

This pull request adds a 'typing_exensions' subproject to 'typing'. The 'typing_extensions' module backports any new additions to 'typing' for Python 3.5+ users who are using older versions of 'typing' that were bundled with their standard library (and so can't update to the latest versions). See python#435 for motivation and additional context.

In retrospect, attempting to backport typing.Collection was a mistake. It's difficult in practice to use it in a consistently meaningful way because the typing module deliberately does not have types like Sequence or List subclass anything resembling the Collections type for older versions of Python. This means that the following code will not typecheck on Python 2 and on older versions of Python 3: from mypy_extensions import Collection def foo(x): # type: Collection[int] -> int return sum(x) foo([1, 2, 3]) ...because `List` is not defined to be a subclass of `Collection` in Python 2 and Python 3.x - 3.5 at runtime and within the typing stubs in typeshed. I think this is potentially repairable with some careful surgery on the typing stubs, but doing that seems highly questionable to me, so I'm thinking attempting to backport `Collection` is out-of-scope for this pull request and should be handled later (or potentially never).

Incorporates change made by python#439, which was recently merged.

Michael0x2a · 2017-07-17T15:42:13Z

@ilevkivskyi -- as an update, I incorporated the changes you just pushed to master earlier today (removing _gorg and _geqv and making other speed improvements).

ilevkivskyi · 2017-07-17T16:10:24Z

@Michael0x2a thanks for continuing to work on this! I will try to make the second review today or tomorrow.

ilevkivskyi · 2017-07-18T16:01:21Z

@Michael0x2a
I responded to your previous comments, continuing with review now.

ilevkivskyi · 2017-07-18T16:08:16Z

Concerning Collection , I would still add it. Maybe not in this PR, but after you make an initial PR to typeshed with typing_extensions stubs.

ilevkivskyi

OK, just three more minor comments.

ilevkivskyi · 2017-07-18T16:28:52Z

typing_extensions/README.rst

+
+The ``typing_extensions`` module contains both backports of changes made to
+the ``typing`` module since Python 3.5.0 as well as experimental types that
+will be eventually added to the ``typing`` module once stabilized.


I think you forgot to mention that it is provisional until Python 3.7.

ilevkivskyi · 2017-07-18T16:33:09Z

typing_extensions/src_py2/test_typing_extensions.py

+        self.assertIsSubclass(collections.Counter, Counter)
+        self.assertIs(type(Counter()), collections.Counter)
+        self.assertIs(type(Counter[T]()), collections.Counter)
+        self.assertIs(type(Counter[int]()), collections.Counter)


I actually wanted to also test that Counter.__bases__ are already erased (before instantiation).

ilevkivskyi · 2017-07-18T16:51:22Z

typing_extensions/src_py3/test_typing_extensions.py

+import typing_extensions
+import collections.abc as collections_abc
+
+TYPING_LATEST = sys.version_info[:3] < (3, 5, 0)


I am not sure I follow the logic here, shouldn't this be TYPING_OLDEST? Or this is the version of typing you "emulate"? Could you please add a comment here?

- Reworded README/description in setup.py to include note about typing's provisional status ending in Python 3.7. - Made the README/description in setup.py a little shorter in general - Added some explanatory comments - Added a test to check Counter.__bases__ is erased in Python 2.7

Michael0x2a · 2017-07-20T06:41:23Z

@ilevkivskyi -- ok, I've addressed your new comments and am ready for the 3rd round of review!

Regarding your comment regarding Counter.__bases__: I think I added a test for that in the Python 3 tests in response to one of your comments from the first round of review, but forgot to add the same test for the Python 2 tests. I copied the same check over.

That said, I want to make sure I actually understand both comments correctly -- are the tests I added what you were looking for, or should I need to add in more checks there?

ilevkivskyi

Maybe there are some small things left but it will be easier to fix them in subsequent smaller PRs, so that I am going to merge this soon. I think the next step is to sumbit typing_extensions stubs to typeshed and probably to "teach" mypy to interact with them (I could imagine some typing types are special-cased in mypy).

Michael0x2a · 2017-07-20T15:14:47Z

@ilevkivskyi stubs for typeshed can be found here: python/typeshed#1471

I'll work on making changes to mypy itself next. iirc it contains some special-casing and tests specifically for mypy_extensions so I'll probably start by duplicating a subset of those for typing_extensions.

See latest comment in python/typing#443 for justification.

the-knights-who-say-ni added the CLA signed label Jun 27, 2017

Michael0x2a changed the title ~~Add typing extensions~~ Add typing extensions [WIP] Jun 27, 2017

Michael0x2a force-pushed the add-typing-extensions branch from 124768e to 6928977 Compare June 27, 2017 09:53

ilevkivskyi reviewed Jun 27, 2017

View reviewed changes

Michael0x2a changed the title ~~Add typing extensions [WIP]~~ Add typing extensions Jun 27, 2017

Michael0x2a force-pushed the add-typing-extensions branch from 6928977 to 15cbbfc Compare June 27, 2017 10:09

ilevkivskyi self-requested a review June 27, 2017 10:14

ilevkivskyi reviewed Jul 7, 2017

View reviewed changes

JelleZijlstra reviewed Jul 7, 2017

View reviewed changes

Michael0x2a mentioned this pull request Jul 7, 2017

Add stubs for typing_extensions module python/typeshed#1471

Merged

Michael0x2a added a commit to Michael0x2a/typeshed that referenced this pull request Jul 8, 2017

Remove Collections backport

bd02b94

See latest comment in python/typing#443 for justification.

Michael0x2a added 8 commits July 17, 2017 08:21

Remove unnecessary gitignore

ed2e142

Fix flake8 errors; make travis test typing_extensions

eb1e5d7

Fix tests and lint errors; make run_tests.py Python 3 only

bded015

Fix typo in .travis.yml

449a339

Try making travis run on each minor version

baa6ba7

Experiment with removing sys.argv hacks

5f55de2

Correctly use sys.version_info

d9d10a5

Michael0x2a added 14 commits July 17, 2017 08:21

Cleanup lingering test changes

2746fae

Address parts of code review

292deb9

Fix tests; tweak setup.py text

11e98ee

Add missing definitions in Python 2 tests

9cc6236

A little closer

ea77bf9

Whoops, forgot old-style classes were a thing

5e243ed

Add check to see if cls.__bases__ is erased

eac513c

Fix import

ea52ad8

Consolidate and document version checks in py3 tests

dd9f31c

Test version check pin

31de561

Special-case quirk with AsyncIterator

67a20cb

Convert readme to RST format; adjust text

6987247

Incorporate speed improvements for typing

11c0fcf

Incorporates change made by python#439, which was recently merged.

Michael0x2a force-pushed the add-typing-extensions branch from eb24e8f to 11c0fcf Compare July 17, 2017 15:36

ilevkivskyi reviewed Jul 18, 2017

View reviewed changes

ilevkivskyi approved these changes Jul 20, 2017

View reviewed changes

ilevkivskyi merged commit 59c0987 into python:master Jul 20, 2017

Michael0x2a added a commit to Michael0x2a/typeshed that referenced this pull request Jul 20, 2017

Remove Collections backport

24e0b82

See latest comment in python/typing#443 for justification.

Michael0x2a added a commit to Michael0x2a/typeshed that referenced this pull request Jul 21, 2017

Remove Collections backport

b9326df

See latest comment in python/typing#443 for justification.

Michael0x2a deleted the add-typing-extensions branch November 17, 2018 05:00

		T_contra = typing.TypeVar('T_contra', contravariant=True) # Ditto contravariant.


		def _gorg(a):



		if __name__ == '__main__':
		# main(argv=[sys.argv[0]] + sys.argv[2:])

Add typing extensions #443

Add typing extensions #443

Conversation

Michael0x2a commented Jun 27, 2017

ilevkivskyi commented Jun 27, 2017

Michael0x2a commented Jun 27, 2017

ilevkivskyi commented Jun 27, 2017

Choose a reason for hiding this comment

Michael0x2a Jun 27, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ilevkivskyi commented Jun 27, 2017

Michael0x2a commented Jun 27, 2017 • edited Loading

ilevkivskyi commented Jun 27, 2017

ilevkivskyi left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ilevkivskyi commented Jul 7, 2017

Michael0x2a commented Jul 7, 2017

gvanrossum commented Jul 7, 2017 via email

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Michael0x2a commented Jul 7, 2017

Michael0x2a commented Jul 8, 2017

Michael0x2a commented Jul 17, 2017

ilevkivskyi commented Jul 17, 2017

ilevkivskyi commented Jul 18, 2017

ilevkivskyi commented Jul 18, 2017

ilevkivskyi left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Michael0x2a commented Jul 20, 2017

ilevkivskyi left a comment

Choose a reason for hiding this comment

Michael0x2a commented Jul 20, 2017

Michael0x2a Jun 27, 2017 •

edited

Loading

Michael0x2a commented Jun 27, 2017 •

edited

Loading