Add support for union types as X | Y (PEP 604)

[cdce8p]

Description

This PR attempts to add the original implementation for PEP 604 proposed by @pprados
The PEP will be added to Python in 3.10. However, since it's already acceptable syntax, we can even use it for older versions with from __future__ import annotations, in type comments or stub files. python/typing#429 (comment)

Example

# python 3.7
from __future__ import annotations
from typing import Union

t1: Union[int, str]
t2: int | str

Test Plan

Tests were included with the original proposal.

Todo

• The original change updates the documentation. Since 3.10 isn't out yet, I would suggest to exclude the doc changes for now.

Edit history

[2020-10-26] Rebased onto master, resolved conflicts
[2020-10-26] Review changes + test fix
[2020-10-26] Removed documentation changes

--
Fixes: #9610

[cdce8p]

@hauntsaninja What is your opinion regarding the documentation?

[hauntsaninja]

My weakly held opinion is that we should leave the docs alone for now. It would be consistent with the docs not using PEP 585 / Python 3.6 is alive and kicking / IMO from __future__ import annotations isn't too widely known. But again, weakly held and only mentioning since you asked me directly :-)

[cdce8p]

My weakly held opinion is that we should leave the docs alone for now. It would be consistent with the docs not using PEP 585 / Python 3.6 is alive and kicking / IMO from __future__ import annotations isn't too widely known. But again, weakly held and only mentioning since you asked me directly :-)

I agree. I'll push a commit removing all references in the docs. If we later decide to keep them, I can just reverse the commit.

[hauntsaninja]

Thanks for this! Looks good to me, but leaving unmerged in case another maintainer has thoughts. Specifically, I'm wondering if it's worth it to try and make this conditional on from __future__ import annotations (knowledge of that unfortunately comes later on in the process).
Also are you @pprados ? If not, we should make sure to add Co-authored-by when merging.

[cdce8p]

@hauntsaninja You might have a point there. Without from __future__ import annotations you get a TypeError. Where would you implement the check? I'm relatively new to mypy development but as far as I see it, I could only add the Python Version check in fastparse.py. Would it be an idea to extend the UnionType to include a flag which we can check at a later point when we have the information about the __future__ flag?

Also are you @pprados ?

No, I'm not. Originally I opened #9610 for it and somebody linked him. However since he didn't respond, I though I open the PR myself to start working on implementing it upstream. I really like the new notation and already started using it in my personal projects together with this mypy implementation 😃

[cdce8p]

I went ahead and added a check for from __future__ import annotations. However, I didn't know where or if I can check for type comments (e.g. # type: int | None). Therefore at the moment those have the same restrictions as normal type declarations: 3.10 or newer OR from __future__ import annotations. The check is not active in stub files.

Regarding the error message: Alternative syntax for unions requires Python 3.10 or newer. This is the same one Pylance and I believe Pyright uses. IMO it's the correct choice not to include the future import here, since technically it will only be fully supported with 3.10 and you should know what you're doing using it for prior versions.

[hauntsaninja]

Thanks, this looks good! I just have some minor nits:

1. Rename the attribute is_binary_op to uses_pep604_syntax
2. I think we should mention it; maybe X | Y syntax for unions requires Python 3.10, but can be used in annotations via PEP 563 on {options.python_version}.

Python 3.9 is going to be around for a long time and it's very much intentional on our part that we're supporting new syntax for it. I wish there was a shorter way to describe from __future__ import annotations that isn't a PEP number, but I do feel like the error message I propose is a not misleading summary of the situation.

I'm not sure what the best way to fix the type error is; I don't know what implications changing SemanticAnalyzerCoreInterface has. I will say that it's not the worst to drop stub file support. Typeshed is conservative in its use of new syntax and worst case one could use the future import in stubs if you wanted to be able to type check Python 3.9

[cdce8p]

@hauntsaninja Thanks for your review.

I think we should mention it; maybe X | Y syntax for unions requires Python 3.10, but can be used in annotations via PEP 563 on {options.python_version}.
Python 3.9 is going to be around for a long time and it's very much intentional on our part that we're supporting new syntax for it. I wish there was a shorter way to describe from __future__ import annotations that isn't a PEP number, but I do feel like the error message I propose is a not misleading summary of the situation.

My reasoning for not adding a note to PEP 563 was that dynamic evaluation of annotations (e.g. with __annotations__ or get_type_hints) doesn't work for versions prior to 3.10, see note PEP 604. If you're fine with that, which I believe many will be, you can use the new syntax without issues. However, if users don't know about this restriction and change their code following the error hint, it might break something. Therefore I would refrain from adding the reference.

Stub files

I might not have described it clearly enough. As far as I know, those will work without issues and you don't even need the future import.

mypy/test-data/unit/check-union-or-syntax.test

Lines 77 to 81 in ae5771e

	[case testUnionOrSyntaxInStubFile]
	# flags: --python-version 3.6
	from lib import x
	[file lib.pyi]
	x: int | None

Type comments

Before, I wasn't sure how I should check for those, since the SemanticAnalyzerCoreInterface doesn't expose it. I would work if we edit fastparse.py -> parse_type_comment to pass an additional flag to visit_BinOp. The drawback is that we either have to pass it to all visit_... functions or add an additional check if it's visit_BinOp. Either way it might cause a small performance hit for large codebases. The alternative would be to support type comments under the same restrictions (future import or 3.10). I implemented the later one for now.

[hauntsaninja]

Some more suggestions:

• Could you add a test with generics, e.g. List[int | str]
• Could you test whether it works when quoted, e.g. def f(x: "int | str"): ...
• Let's move is_type_comment to being an attribute of TypeConverter, rather than changing visit.
• I think you'll need to set is_type_comment for functions somewhere else in fastparse. Might be worth grepping through wherever we instantiate TypeConverter
• pep604_syntax.uses_pep604_syntax is a little bit of a mouthful. Maybe rename to syntax.uses_pep604 or syntax_form.uses_pep604? Actually, maybe the right thing to do is un-nest it and just have two attributes: uses_pep604 and is_evaluated, where is_evaluated is True if it's not a type comment or string literal. is_evaluated is something we could generalise beyond UnionType, eg for Detect if required string literal escapes are missing from a type #948 or for helping guide PEP 585 usage.

[hauntsaninja]

And okay, we can leave out the PEP 563 mention. You're right that it's the more conservative change; we can always reconsider if we find people are confused :-)

[cdce8p]

@hauntsaninja Thanks again. Regarding your points:

Could you test whether it works when quoted, e.g. def f(x: "int | str"): ...

Good catch. Turns out that it wouldn't have worked. I also added a test case for generics.

Let's move is_type_comment to being an attribute of TypeConverter, rather than changing visit

Great idea. I should have though of that. It just makes much more sense.

pep604_syntax.uses_pep604_syntax

I've changed it to just use uses_pep604_syntax and is_evaluated as you suggested. I wasn't sure if I should really add is_evaluated to the Type base class, especially since I just use it for this change and haven't tested any other case. Therefore I added it to UnionType.

I was thinking, we don't have any documentation for this change. Although, as discussed, replacing Union throughout the docs isn't great, should I add least add a Paragraph somewhere describing the possibility for the new syntax and stating the limitation?

[hauntsaninja]

Thanks, this looks great!

One last nit: I think the is_type_comment attribute on TypeConverter should be renamed to is_evaluated, since we call parse_type_comment from parse_type_string.

Documentation is a good idea! https://mypy.readthedocs.io/en/stable/kinds_of_types.html#union-types seems like the natural place

[cdce8p]

@hauntsaninja I changed the attribute name and added a few more test cases. The commit also includes my first draft for the documentation. Let me know what you think :)

[cdce8p]

Any updates? What is left to do before it can be merged?

[hauntsaninja]

Documentation looks good, but might be a little long for the page it's on. Should probably move the details onto a separate page that also discusses PEP 585 support. But we can do that in another PR. Thanks again! :-)

[cdce8p]

@hauntsaninja Thanks for the reviews. Those changes really improved this PR!

[jni]

And okay, we can leave out the PEP 563 mention. You're right that it's the more conservative change; we can always reconsider if we find people are confused :-)

fwiw, @stefanv and I were indeed having a confused conversation about this on the scikit-image Zulip chat 😂 (needs sign-up/login, can use GH login). TLDR, thank you both for implementing this, it is just sooooo much nicer to use this syntax, but yes, it would be great to have a better place than this issue to tell people, hey, you can use this today in 3.7+ with from __future__ import annotations. (Note that most of the scientific Python community has moved to 3.7+ thanks to this document.)

In short, we'd favour some documentation around this and I'm happy to take a stab at a short sentence/paragraph if someone points me to where they'd like it to go. =) Thank you again!

[hauntsaninja]

I changed the documentation already. There's a new section that should help with this sort of question: https://mypy.readthedocs.io/en/stable/runtime_troubles.html.
It mentions X | Y syntax here: https://mypy.readthedocs.io/en/stable/runtime_troubles.html#using-x-y-syntax-for-unions

[jni]

Lovely, thank you! 🙏