Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Complete type analysis of variadic types #15991

Merged
merged 5 commits into from
Sep 7, 2023

Conversation

ilevkivskyi
Copy link
Member

@ilevkivskyi ilevkivskyi commented Aug 29, 2023

This PR closes the first part of support for TypeVarTuple: the "static" analysis of types (of course everything is static in mypy, but some parts are more static): semanal/typeanal, expand_type(), map_instance_to_supertype(), erase_type() (things that precede and/or form foundation for type inference and subtyping). This one was quite tricky, supporting unpacks of forward references required some thinking.

What is included in this PR:

  • Moving argument count validation from semanal_typeargs to typeanal. In one of previous PRs I mentioned that get_proper_type() may be called during semantic analysis causing troubles if we have invalid aliases. So we need to move validation to early stage. For instances, this is not required, but I strongly prefer keeping instances and aliases similar. And ideally at some point we can combine the logic, since it gets more and more similar. At some point we may want to prohibit using get_proper_type() during semantic analysis, but I don't want to block TypeVarTuple support on this, since this may be a significant refactoring.
  • Fixing map_instance_to_supertype() and erase_type(). These two are straightforward, we either use expand_type() logic directly (by calling it), or following the same logic.
  • Few simplifications in expandtype and typeops following previous normalizations of representation, unless there is a flaw in my logic, removed branches should be all dead code.
  • Allow (only fixed) unpacks in argument lists for non-variadic types. They were prohibited for no good reason.
  • (Somewhat limited) support for forward references in unpacks. As I mentioned this one is tricky because of how forward references are represented. Usually they follow either a life cycle like: Any -> <known type>, or <Any> -> <placeholder> -> <known type> (second one is relatively rare and usually only appears for potentially recursive things like base classes or type alias targets). It looks like <placeholder> can never appear as a valid unpack target, I don't have a proof for this, but I was not able to trigger this, so I am not handling it (possible downside is that there may be extra errors about invalid argument count for invalid unpack targets). If I am wrong and this can happen in some valid cases, we can add handling for unpacks of placeholders later. Currently, the handling for Any stage of forward references is following: if we detect it, we simply create a dummy valid alias or instance. This logic should work for the same reason having plain Any worked in the first place (and why all tests pass if we delete visit_placeholder_type()): because (almost) each time we analyze a type, it is either already complete, or we analyze it from scratch, i.e. we call expr_to_unanalyzed_type(), then visit_unbound_type() etc. We almost never store "partially analyzed" types (there are guards against incomplete references and placeholders in annotations), and when we do, it is done in a controlled way that guarantees a type will be re-analyzed again. Since this is such a tricky subject, I didn't add any complex logic to support more tricky use cases (like multiple forward references to fixed unpacks in single list). I propose that we release this, and then see what kind of bug reports we will get.
  • Additional validation for type arguments position to ensure that TypeVarTuples are never split. Total count is not enough to ban case where we have type variables [T, *Ts, S, U] and arguments [int, int, *Us, int]. We need to explicitly ensure that actual suffix and prefix are longer or equal to formal ones. Such splitting would be very hard to support, and is explicitly banned by the PEP.
  • Few minor cleanups.

Some random comments:

  • It is tricky to preserve valid parts of type arguments, if there is an argument count error involving an unpack. So after such error I simply set all arguments to Any (or *tuple[Any, ...] when needed).
  • I know there is some code duplication. I tried to factor it away, but it turned out non-trivial. I may do some de-duplication pass after everything is done, and it is easier to see the big picture.
  • Type applications (i.e. when we have A[int, int] in runtime context) are wild west currently. I decided to postpone variadic support for them to a separate PR, because there is already some support (we will just need to handle edge cases and more error conditions) and I wanted minimize size of this PR.
  • Something I wanted to mention in one of previous PRs but forgot: Long time ago I proposed to normalize away type aliases inside Unpack, but I abandoned this idea, it doesn't really give us any benefits.

As I said, this is the last PR for the "static part", in the next PR I will work on fixing subtyping and inference for variadic instances. And then will continue with remaining items I mentioned in my master plan in #15924

Fixes #15978

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@mehdigmira
Copy link

FYI, this PR seems to be introducing some regressions, see example below

from typing import Generic
from typing_extensions import TypeVarTuple
from typing import Unpack

Ts = TypeVarTuple("Ts")


class Array(Generic[Unpack[Ts]]):
    def _close(self) -> None:
        ...

    def close(self) -> None:
        self._close()  # error: Invalid self argument "Array[Unpack[Ts]]" to attribute function "_close" with type "Callable[[Array[Unpack[Ts]]], None]"

@ilevkivskyi
Copy link
Member Author

@mehdigmira This is kind of expected. Subtyping is quite broken now (especially for instances), and this PR exposes this. As I mentioned in PR description, I am not going to work on subtyping in this PR, I am going to fix it in the next one(s).

@github-actions
Copy link
Contributor

github-actions bot commented Sep 2, 2023

According to mypy_primer, this change doesn't affect type check results on a corpus of open source code. ✅

Copy link
Collaborator

@JukkaL JukkaL left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good! This fixes a bunch of important issues with TypeVarTuples.

Maybe reword the beginning of the summary, since the initial "Fixes #..." kind of downplays the impact of this PR, as it fixes other issues as well.

@ilevkivskyi
Copy link
Member Author

@JukkaL OK 👍 I updated the PR description. (Btw the next part is almost ready, I will make a PR likely tomorrow.)

@ilevkivskyi ilevkivskyi merged commit 8b73cc2 into python:master Sep 7, 2023
18 checks passed
@ilevkivskyi ilevkivskyi deleted the variadic-type-analisys branch September 7, 2023 22:23
ilevkivskyi added a commit that referenced this pull request Sep 13, 2023
The second part of support for user defined variadic types comes as a
single PR, it was hard to split into smaller parts. This part covers
subtyping and inference (and relies on the first part: type analysis,
normalization, and expansion, concluded by
#15991). Note btw that the third (and
last) part that covers actually using all the stuff in `checkexpr.py`
will likely come as several smaller PRs.

Some comments on this PR:
* First good news: it looks like instances subtyping/inference can be
handled in a really simple way, we just need to find correct type
arguments mapping for each type variable, and perform procedures
argument by argument (note this heavily relies on the normalization).
Also callable subtyping inference for variadic items effectively defers
to corresponding tuple types. This way all code paths will ultimately go
through variadic tuple subtyping/inference (there is still a bunch of
boilerplate to do the mapping, but it is quite simple).
* Second some bad news: a lot of edge cases involving `*tuple[X, ...]`
were missing everywhere (even couple cases in the code I touched
before). I added all that were either simple or important. We can handle
more if users will ask, since it is quite tricky.
* Note that I handle variadic tuples essentially as infinite unions, the
core of the logic for this (and for most of this PR FWIW) is in
`variadic_tuple_subtype()`.
* Previously `Foo[*tuple[int, ...]]` was considered a subtype of
`Foo[int, int]`. I think this is wrong. I didn't find where this is
required in the PEP (see one case below however), and mypy currently
considers `tuple[int, ...]` not a subtype of `tuple[int, int]` (vice
versa are subtypes), and similarly `(*args: int)` vs `(x: int, y: int)`
for callables. Because of the logic I described in the first comment,
the same logic now uniformly applies to instances as well.
* Note however the PEP requires special casing of `Foo[*tuple[Any,
...]]` (equivalent to bare `Foo`), and I agree we should do this. I
added a minimal special case for this. Note we also do this for
callables as well (`*args: Any` is very different from `*args: object`).
And I think we should special case `tuple[Any, ...] <: tuple[int, int]`
as well. In the future we can even extend the special casing to
`tuple[int, *tuple[Any, ...], int]` in the spirit of
#15913
* In this PR I specifically only handle the PEP required item from above
for instances. For plain tuples I left a TODO, @hauntsaninja may
implement it since it is needed for other unrelated PR.
* I make the default upper bound for `TypeVarTupleType` to be
`tuple[object, ...]`. I think it can never be `object` (and this
simplifies some subtyping corner cases).
* TBH I didn't look into callables subtyping/inference very deeply
(unlike instances and tuples), if needed we can improve their handling
later.
* Note I remove some failing unit tests because they test non-nomralized
forms that should never appear now. We should probably add some more unit
tests, but TBH I am quite tired now.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

crash when using isinstance againt variadic tuple
4 participants