Supporting type_comments=True

[gvanrossum]

The ast.parse() function has two barely documented features, type_comments= and feature_version=, which modify the tokenizer and parser for the benefit of (primarily) mypy. type_comments=True changes the tokenizer to return TYPE_COMMENT tokens when it detects a type comment (a comment starting with 'type:'), and the old Grammar file has [TYPE_COMMENT] sprinkled throughout to support these.

In addition, feature_version=(3, N) enables a few changes in the tokenizer and ast.c to support older Python versions (mostly related to the three different stages of support for async and await keywords). [moved to #124]

We'll have to support these. I think it won't be very difficult since most of the logic is part of the tokenizer, but we'll have to check ast.c for other tests for feature_version. Note that these flags are passed in different forms to compile(), which has the actual support, but undocumented. See the source of ast.py.

[gvanrossum]

IMO for alpha6 we can just make these flags use the old parser. But we do need to do that (else mypy won't be able to work with alpha 6 unless you use -X oldparser).

[lysnikolaou]

IMO for alpha6 we can just make these flags use the old parser. But we do need to do that (else mypy won't be able to work with alpha 6 unless you use -X oldparser).

Done in 0bd4bc3, which needs review though.

[gvanrossum]

Okay, I am looking into how to do this post-alpha6. (FWIW I've fixed mypy to work with the 3.9 but you need to use master from https://github.com/python/mypy/.)

There are actually several separate things to implement:

• support for type comments in the grammar other than long-form arguments
• long-form argument type comments (this looks very hairy)
• support for # type: ignore
• support for func_type_input