New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor reversible operators #5475

Merged
merged 4 commits into from Aug 16, 2018

Conversation

Projects
None yet
3 participants
@Michael0x2a
Collaborator

Michael0x2a commented Aug 14, 2018

This pull request refactors and reworks how we handle reversible operators like __add__.

Specifically, what our code was previously doing was assuming that given the expression A() + B(), we would always try calling A().__add__(B()) first, followed by B().__radd__(A()) second (if the __radd__ method exists).

Unfortunately, it seems like this model was a little too naive, which caused several mismatches/weird errors when I was working on refining how we handle overlaps and TypeVars in a subsequent PR.

Specifically, what actually happens is that...

  1. When doing A() + A(), we only ever try calling A.__add__, never A.__radd__. This is the case even if __add__ is undefined.

  2. If B is a subclass of A, and if B defines an __radd__ method, and we do A() + B(), Python will actually try checking B.__radd__ first, then A.__add__ second.

    This lets a subclass effectively "refine" the desired return type.

    Note that if B only inherits an __radd__ method, Python calls A.__add__ first as usual. Basically, B must provide a genuine refinement over whatever A returns.

  3. In all other cases, we call __add__ then __radd__ as usual.

This pull request modifies both checker.py and checkexpr.py to match this behavior, and adds logic so that we check the calls in the correct order.

This ended up slightly changing a few error messages in certain edge cases.

@Michael0x2a Michael0x2a requested a review from ilevkivskyi Aug 14, 2018

Michael0x2a added a commit to Michael0x2a/mypy that referenced this pull request Aug 14, 2018

Add support for partially overlapping types
This pull request adds more robust support for detecting partially
overlapping types. Specifically, it detects overlaps with...

1. TypedDicts
2. Tuples
3. Unions
4. TypeVars
5. Generic types containing variations of the above.

It also swaps out the code for detecting overlaps with operators and
removes some associated (and now unused) code.

This pull request builds on top of python#5474
and python#5475 -- once those two PRs are
merged, I'll rebase this diff if necessary.

This pull request also supercedes python#5475 --
that PR contains basically the same code as these three PRs, just
smushed together.
Show outdated Hide outdated mypy/nodes.py Outdated
Refactor reversible operators
This pull request refactors and reworks how we handle reversible
operators like __add__.

Specifically, what our code was previously doing was assuming that given
the expression `A() + B()`, we would always try calling
`A().__add__(B())` first, followed by `B().__radd__(A())` second (if the
`__radd__` method exists).

Unfortunately, it seems like this model was a little too naive, which
caused several mismatches/weird errors when I was working on refining
how we handle overlaps and TypeVars in a subsequent PR.

Specifically, what actually happens is that...

1. When doing `A() + A()`, we only ever try calling `A.__add__`, never
   `A.__radd__`. This is the case even if `__add__` is undefined.

2. If `B` is a subclass of `A`, and if `B` defines an `__radd__` method,
   and we do `A() + B()`, Python will actually try checking `B.__radd__`
   *first*, then `A.__add__` second.

   This lets a subclass effectively "refine" the desired return type.

   Note that if `B` only *inherits* an `__radd__` method, Python calls
   `A.__add__` first as usual. Basically, `B` must provide a genuine
   refinement over whatever `A` returns.

3. In all other cases, we call `__add__` then `__radd__` as usual.

This pull request modifies both checker.py and checkexpr.py to match
this behavior, and adds logic so that we check the calls in the
correct order.

This ended up slightly changing a few error messages in certain edge
cases.

Michael0x2a added a commit to Michael0x2a/mypy that referenced this pull request Aug 14, 2018

Add support for partially overlapping types
This pull request adds more robust support for detecting partially
overlapping types. Specifically, it detects overlaps with...

1. TypedDicts
2. Tuples
3. Unions
4. TypeVars
5. Generic types containing variations of the above.

It also swaps out the code for detecting overlaps with operators and
removes some associated (and now unused) code.

This pull request builds on top of python#5474
and python#5475 -- once those two PRs are
merged, I'll rebase this diff if necessary.

This pull request also supercedes python#5475 --
that PR contains basically the same code as these three PRs, just
smushed together.
@ilevkivskyi

Thanks! Looks good, I have several comments, all of them are minor.

Show outdated Hide outdated mypy/checker.py Outdated
Show outdated Hide outdated mypy/checker.py Outdated
Show outdated Hide outdated mypy/checker.py Outdated
Show outdated Hide outdated mypy/checker.py Outdated
Show outdated Hide outdated mypy/checker.py Outdated
Show outdated Hide outdated mypy/checkexpr.py Outdated
errors.append(local_errors)
results.append(result)
else:
return result

This comment has been minimized.

@ilevkivskyi

ilevkivskyi Aug 14, 2018

Collaborator

I am not sure I understand how can this ever succeed, could you please add a comment with an example?

@ilevkivskyi

ilevkivskyi Aug 14, 2018

Collaborator

I am not sure I understand how can this ever succeed, could you please add a comment with an example?

This comment has been minimized.

@Michael0x2a

Michael0x2a Aug 16, 2018

Collaborator

I originally added that out just in case, but it seems we do fall into that case at least once -- for example, adding an assert there makes this test case fail.

I added a TODO.

@Michael0x2a

Michael0x2a Aug 16, 2018

Collaborator

I originally added that out just in case, but it seems we do fall into that case at least once -- for example, adding an assert there makes this test case fail.

I added a TODO.

Show outdated Hide outdated mypy/messages.py Outdated
Show outdated Hide outdated test-data/unit/check-classes.test Outdated
Show outdated Hide outdated test-data/unit/check-classes.test Outdated

Michael0x2a added some commits Aug 16, 2018

@ilevkivskyi

Great, thanks! You can merge this after you fix the last few comments here. Then rebase the overlap PR and I will review it.

# Currently, it seems we still need this to correctly deal with
# things like metaclasses?
#
# E.g. see the pythoneval.testMetaclassOpAccessAny test case.

This comment has been minimized.

@ilevkivskyi

ilevkivskyi Aug 16, 2018

Collaborator

Could you please add details about this to #5136?

@ilevkivskyi

ilevkivskyi Aug 16, 2018

Collaborator

Could you please add details about this to #5136?

# This is probably related to the TODO in lookup_operator(...)
# up above.
#
# TODO: Remove this extra case

This comment has been minimized.

@ilevkivskyi

ilevkivskyi Aug 16, 2018

Collaborator

I think this is something deserving a follow-up issue. Could you please open one?

@ilevkivskyi

ilevkivskyi Aug 16, 2018

Collaborator

I think this is something deserving a follow-up issue. Could you please open one?

# gracefully -- it doesn't correctly switch to using __truediv__ when
# 'from __future__ import division' is included, it doesn't display a very
# graceful error if __div__ is missing but __truediv__ is present...
# Also see https://github.com/python/mypy/issues/2048

This comment has been minimized.

@ilevkivskyi

ilevkivskyi Aug 16, 2018

Collaborator

Could you please add these details to the issue and raise its priority to at least normal? This future import is very important in loads of numeric code.

@ilevkivskyi

ilevkivskyi Aug 16, 2018

Collaborator

Could you please add these details to the issue and raise its priority to at least normal? This future import is very important in loads of numeric code.

Show outdated Hide outdated mypy/checkexpr.py Outdated

@Michael0x2a Michael0x2a merged commit 7f29b73 into python:master Aug 16, 2018

2 checks passed

continuous-integration/appveyor/pr AppVeyor build succeeded
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details

Michael0x2a added a commit to Michael0x2a/mypy that referenced this pull request Aug 16, 2018

Add support for partially overlapping types
This pull request adds more robust support for detecting partially
overlapping types. Specifically, it detects overlaps with...

1. TypedDicts
2. Tuples
3. Unions
4. TypeVars
5. Generic types containing variations of the above.

It also swaps out the code for detecting overlaps with operators and
removes some associated (and now unused) code.

This pull request builds on top of python#5474
and python#5475 -- once those two PRs are
merged, I'll rebase this diff if necessary.

This pull request also supercedes python#5475 --
that PR contains basically the same code as these three PRs, just
smushed together.

@Michael0x2a Michael0x2a deleted the Michael0x2a:refactor-reversible-operators branch Aug 27, 2018

Michael0x2a added a commit that referenced this pull request Aug 27, 2018

Add partial overload checks (#5476)
This pull request adds more robust support for detecting partially
overlapping types. Specifically, it detects overlaps with...

1. TypedDicts
2. Tuples
3. Unions
4. TypeVars
5. Generic types containing variations of the above.

This new overlapping type check is used for detecting unsafe overlaps
with overloads and operator methods as well as for performing 
reachability/unreachability analysis in `isinstance` and `if x is None`
checks and the like.

This PR also removes some (now unused) code that used to be used
for detecting overlaps with operators.

This pull request builds on top of #5474
and #5475 and supersedes 
#5475.

Michael0x2a added a commit to Michael0x2a/mypy that referenced this pull request Sep 4, 2018

Remove the 'rop will not be called' error message
python#5475 introduced a new type of error
message ("__rop__ will not be called when evaluating 'a + b'...") that
triggers when the user tries evaluating expressions like `foo + foo`
where `foo` does not contain an `__add__` method that accepts a value
of the same type, but *does* contain an `__radd__` method that does.

This pull request removes that error message on the grounds that it's
too cryptic and unlikely to be helpful to most mypy users. That error
message is useful mainly for people developing libraries containing
custom numeric types (or libraries that appropriate operators to create
custom DSLs) -- however, most people are not library creators and so
will not find this error message useful.

ilevkivskyi added a commit that referenced this pull request Sep 5, 2018

Remove the 'rop will not be called' error message (#5571)
#5475 introduced a new type of error
message ("__rop__ will not be called when evaluating 'a + b'...") that
triggers when the user tries evaluating expressions like `foo + foo`
where `foo` does not contain an `__add__` method that accepts a value
of the same type, but *does* contain an `__radd__` method that does.

This pull request removes that error message on the grounds that it's
too cryptic and unlikely to be helpful to most mypy users. That error
message is useful mainly for people developing libraries containing
custom numeric types (or libraries that appropriate operators to create
custom DSLs) -- however, most people are not library creators and so
will not find this error message useful.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment