Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ignore position if imprecise arguments are matched by name #16471

Merged
merged 5 commits into from
Nov 13, 2023

Conversation

ilevkivskyi
Copy link
Member

Fixes #16405
Fixes #16412

Imprecise argument kinds inference was added a while ago to support various edge cases with ParamSpec. This feature required mapping actual kinds to formal kinds, which is in general undecidable. At that time we decided to not add much special-casing, and wait for some real use-cases. So far there are two relevant issues, and it looks like both of them can be fixed with simple special-casing: ignore argument positions in subtyping if arguments can be matched by name. This adds minor unsafety, and generally doesn't look bad, so I think we should go ahead with it.

This comment has been minimized.

Copy link
Member

@AlexWaygood AlexWaygood left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems reasonable!

This adds minor unsafety

Could you give some examples of where this would make things less safe? Not sure I fully understand the trade-off here

mypy/subtypes.py Show resolved Hide resolved
def test(x: int) -> int: ...
apply(apply, test, x=42) # OK
apply(apply, test, 42) # Also OK (but requires some special casing)
apply(apply, test, "bad") # E: Argument 1 to "apply" has incompatible type "Callable[[Callable[P, T], **P], None]"; expected "Callable[[Callable[[int], int], str], None]"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Off-topic for this PR, and I'm guessing this might be tricky to implement, but it would be really nice if mypy could complain about argument 2 in this error message rather than argument 1. In real-world code, the "mistake" in calls like this is generally passing the wrong kinds of object as arguments after the function, rather than passing in the wrong function altogether

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, this is really hard.

test-data/unit/check-parameter-specification.test Outdated Show resolved Hide resolved
backend="asyncio",
))
submit(
run, # E: Argument 1 to "submit" has incompatible type "Callable[[Callable[[], T_Retval], VarArg(object), DefaultNamedArg(str, 'backend')], T_Retval]"; expected "Callable[[Callable[[], Result], int], Result]"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also off-topic for this PR, but I find it unfortunate that we use these mypy_extensions types in error messages, given that we document them as deprecated :(

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I actually was thinking about the same thing myself as I was looking at this test. This is not hard, but tedious. It would be great if someone will do this.

test-data/unit/check-parameter-specification.test Outdated Show resolved Hide resolved
@AlexWaygood
Copy link
Member

AlexWaygood commented Nov 12, 2023

The primer output looks really good.

2/4 of the primer hits are places where people are passing overloaded functions (such as builtins.open or tempfile.mkdtemp) as arguments to higher-order functions that are annotated using ParamSpecs (such as concurrent.futures.ThreadPoolExecutor.submit). Doesn't look like we have many tests for that kind of thing -- worth adding some?

ilevkivskyi and others added 2 commits November 12, 2023 20:33
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
@ilevkivskyi
Copy link
Member Author

Doesn't look like we have many tests for that kind of thing -- worth adding some?

In principle yes, but this is not really related to this PR.

@ilevkivskyi
Copy link
Member Author

Could you give some examples of where this would make things less safe? Not sure I fully understand the trade-off here

I actually didn't try, but I think there may be false negatives where two actual arguments are mapped to the same formal argument in the callable if they have the same type, and one is positional, and the second is named.

@AlexWaygood
Copy link
Member

Could you give some examples of where this would make things less safe? Not sure I fully understand the trade-off here

I actually didn't try, but I think there may be false negatives where two actual arguments are mapped to the same formal argument in the callable if they have the same type, and one is positional, and the second is named.

Okay, I think I see (though I'm still struggling to think up an actual example of code where this would cause an issue). In any case, that seems like very much an edge case, so I agree that it's better to prioritise avoiding false positive here

Copy link
Member

@AlexWaygood AlexWaygood left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@ilevkivskyi ilevkivskyi mentioned this pull request Nov 12, 2023
2 tasks

This comment has been minimized.

Copy link
Collaborator

@JukkaL JukkaL left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems reasonable. Avoiding false positives here seems more important than a small risk of false negatives.

Copy link
Contributor

Diff from mypy_primer, showing the effect of this PR on open source code:

pytest (https://github.com/pytest-dev/pytest)
+ src/_pytest/_py/path.py:763: error: Unused "type: ignore" comment  [unused-ignore]
+ src/_pytest/_py/path.py:1280: error: Unused "type: ignore" comment  [unused-ignore]

poetry (https://github.com/python-poetry/poetry)
+ tests/utils/test_cache.py:345: error: Unused "type: ignore" comment  [unused-ignore]

anyio (https://github.com/agronholm/anyio)
+ src/anyio/from_thread.py:395: error: Unused "type: ignore" comment  [unused-ignore]

discord.py (https://github.com/Rapptz/discord.py): typechecking got 1.07x faster (171.9s -> 160.2s)
(Performance measurements are based on a single noisy sample)

@ilevkivskyi ilevkivskyi merged commit c6cb3c6 into python:master Nov 13, 2023
18 checks passed
@ilevkivskyi ilevkivskyi deleted the tune-paramspec-inference-kind branch November 13, 2023 16:59
JukkaL pushed a commit that referenced this pull request Nov 22, 2023
Fixes #16405
Fixes #16412

Imprecise argument kinds inference was added a while ago to support
various edge cases with `ParamSpec`. This feature required mapping
actual kinds to formal kinds, which is in general undecidable. At that
time we decided to not add much special-casing, and wait for some real
use-cases. So far there are two relevant issues, and it looks like both
of them can be fixed with simple special-casing: ignore argument
positions in subtyping if arguments can be matched by name. This adds
minor unsafety, and generally doesn't look bad, so I think we should go
ahead with it.

---------

Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
3 participants