Enable new type inference by default

[ilevkivskyi]

Fixes #15906

I am adding --old-type-inference so people can disable the flag if they have issues (for few releases). IIRC there will be some fallback in mypy_primer, but last time I checked it was all correct. Also I don't remember if we need to update some tests, but we will see.

[JukkaL]

I checked --new-type-inference with our big internal codebase, and there were a few dozen additional errors. None of them looked serious, and they don't block the switch.

Here are the main differences, just as a FYI (no need to fix these, though I may mention some of these in the release blog post):

• With strict optional disabled, some decorators started stripping away optional types. I remember that we've seen this before, and this is essentially a side effect of disabling strict optional checking. Ensuring that we preserve optional types would be hard (and perhaps unprincipled), so this seems acceptable.
• Some error messages were changed, and some # type: ignore comments had to be adjusted.
• Some code that previously didn't generate errors due to unsolvable type variables started generating errors. At least in some cases we used to infer unbound type variables, and now generate an error, which seems like an improvement.
• There were two cases where type inference failed because of trying to infer against an overloaded generic callable. Previously this didn't generate an error, but I think we leaked type variables, so code was passing type checking previously by accident, I think.

[ilevkivskyi]

@JukkaL Thanks! It looks like there are two new errors related to overloads in the mypy_primer output, everything else looks good. I will double-check what is going on with overloads.

[ilevkivskyi]

OK, so the errors in mypy_primer were not caused by overloads (it was just a coincidence), the errors were caused by lambdas. Namely, we usually silence the type errors when accepting arguments for the purpose of inferring callee type variables (because type context may be still incomplete). Accidentally, the silencing logic did not cover the second pass of inference. Before this was OK, because we used to provide a heavily erased context to lambdas. But with the new inference algorithm we provide a richer context to lambdas, which now could cause errors if type context is still incomplete (e.g. if we can infer type variable from one lambda, and use it for the second lambda, as it happened in both cases in mypy_primer)

[ilevkivskyi]

I decided to add a dedicated test for the scenario I described above, since it looks like a relatively common pattern.

[ilevkivskyi]

Oh wow, took me almost an hour to figure out why I can't reproduce locally the "false negative" in prefect. It turns out that now we give the errors only once, while before the Unexpected keyword argument errors were given twice, which is really weird. I guess there is a flaw in error de-duplication logic related to "foo" defined here notes.

Anyway, I checked all the suspicious things in mypy_primer again. All looks good now!

[github-actions]

Diff from mypy_primer, showing the effect of this PR on open source code:

openlibrary (https://github.com/internetarchive/openlibrary)
+ openlibrary/catalog/utils/__init__.py:19: error: Unused "type: ignore" comment  [unused-ignore]

pandas (https://github.com/pandas-dev/pandas): typechecking got 1.05x slower (420.4s -> 442.4s)
(Performance measurements are based on a single noisy sample)
+ pandas/core/frame.py:6846: error: Unused "type: ignore" comment  [unused-ignore]

prefect (https://github.com/PrefectHQ/prefect)
- src/prefect/_internal/concurrency/calls.py:267: error: Incompatible return value type (got "Task[_T]", expected "Awaitable[T] | None")  [return-value]
- src/prefect/_internal/concurrency/calls.py:272: error: Incompatible return value type (got "_T", expected "Awaitable[T] | None")  [return-value]
- src/prefect/_internal/concurrency/api.py:167: note:          abstractstaticmethod[[Callable[[], T] | Call[T], float | None, Iterable[Call[Any]] | None], T]
+ src/prefect/_internal/concurrency/api.py:167: note:          abstractstaticmethod[[Callable[[], _R_co] | Call[_R_co], float | None, Iterable[Call[Any]] | None], _R_co]
- src/prefect/_internal/concurrency/api.py:185: note:          abstractstaticmethod[[Callable[[], T] | Call[T], float | None, Iterable[Call[Any]] | None], T]
+ src/prefect/_internal/concurrency/api.py:185: note:          abstractstaticmethod[[Callable[[], _R_co] | Call[_R_co], float | None, Iterable[Call[Any]] | None], _R_co]
- src/prefect/_internal/concurrency/api.py:225: note:          abstractstaticmethod[[Callable[[], T] | Call[T], float | None, Iterable[Call[Any]] | None], T]
+ src/prefect/_internal/concurrency/api.py:225: note:          abstractstaticmethod[[Callable[[], _R_co] | Call[_R_co], float | None, Iterable[Call[Any]] | None], _R_co]
- src/prefect/_internal/concurrency/api.py:246: note:          abstractstaticmethod[[Callable[[], T] | Call[T], float | None, Iterable[Call[Any]] | None], T]
+ src/prefect/_internal/concurrency/api.py:246: note:          abstractstaticmethod[[Callable[[], _R_co] | Call[_R_co], float | None, Iterable[Call[Any]] | None], _R_co]
- src/prefect/engine.py:2371: error: Unexpected keyword argument "task" for "create_call"  [call-arg]
- src/prefect/_internal/concurrency/api.py:38: note: "create_call" defined here
- src/prefect/engine.py:2371: error: Unexpected keyword argument "task_run" for "create_call"  [call-arg]
- src/prefect/_internal/concurrency/api.py:38: note: "create_call" defined here
- src/prefect/engine.py:2371: error: Unexpected keyword argument "state" for "create_call"  [call-arg]
- src/prefect/_internal/concurrency/api.py:38: note: "create_call" defined here
- src/prefect/engine.py:2419: error: Unexpected keyword argument "flow" for "create_call"  [call-arg]
- src/prefect/_internal/concurrency/api.py:38: note: "create_call" defined here
- src/prefect/engine.py:2419: error: Unexpected keyword argument "flow_run" for "create_call"  [call-arg]
- src/prefect/_internal/concurrency/api.py:38: note: "create_call" defined here
- src/prefect/engine.py:2419: error: Unexpected keyword argument "state" for "create_call"  [call-arg]
- src/prefect/_internal/concurrency/api.py:38: note: "create_call" defined here
- src/prefect/runner/runner.py:1153: error: Unexpected keyword argument "flow" for "create_call"  [call-arg]
- src/prefect/_internal/concurrency/api.py:38: note: "create_call" defined here
- src/prefect/runner/runner.py:1153: error: Unexpected keyword argument "flow_run" for "create_call"  [call-arg]
- src/prefect/_internal/concurrency/api.py:38: note: "create_call" defined here
- src/prefect/runner/runner.py:1153: error: Unexpected keyword argument "state" for "create_call"  [call-arg]
- src/prefect/_internal/concurrency/api.py:38: note: "create_call" defined here

Tanjun (https://github.com/FasterSpeeding/Tanjun)
- tanjun/dependencies/reloaders.py:366: error: Argument 1 to "filter" has incompatible type "Callable[[Path], bool]"; expected "Callable[[Self], TypeGuard[bool]]"  [arg-type]
- tanjun/dependencies/reloaders.py:366: error: Argument 1 to "map" has incompatible type "Callable[[Self, bool], Self]"; expected "Callable[[Path], Self]"  [arg-type]

mypy (https://github.com/python/mypy)
+ Warning: --new-type-inference flag is deprecated; new type inference algorithm is already enabled by default

jax (https://github.com/google/jax)
+ jax/_src/tree_util.py:343: error: Unused "type: ignore[call-overload, index]" comment  [unused-ignore]
+ jax/_src/tree_util.py:438: error: Unused "type: ignore" comment  [unused-ignore]
+ jax/_src/state/utils.py:32: error: Need type annotation for "const_avals"  [var-annotated]
+ jax/_src/nn/functions.py:454: error: Unused "type: ignore" comment  [unused-ignore]
+ jax/_src/third_party/scipy/interpolate.py:154: error: Unused "type: ignore" comment  [unused-ignore]
+ jax/_src/third_party/scipy/interpolate.py:156: error: Unused "type: ignore" comment  [unused-ignore]
+ jax/example_libraries/optimizers.py:120: error: Unused "type: ignore" comment  [unused-ignore]
+ jax/experimental/sparse/bcoo.py:864: error: Unused "type: ignore" comment  [unused-ignore]
+ jax/experimental/sparse/bcoo.py:897: error: Unused "type: ignore" comment  [unused-ignore]
+ jax/experimental/jax2tf/jax2tf.py:2991: error: Unused "type: ignore" comment  [unused-ignore]

pytest (https://github.com/pytest-dev/pytest)
+ src/_pytest/_py/path.py:319: error: Unused "type: ignore[arg-type, return-value]" comment  [unused-ignore]

scikit-learn (https://github.com/scikit-learn/scikit-learn)
+ sklearn/neighbors/tests/test_neighbors.py:83: error: Unused "type: ignore" comment  [unused-ignore]

ibis (https://github.com/ibis-project/ibis)
- ibis/expr/types/relations.py:3971: error: Argument 1 to "map" has incompatible type "Callable[[V | Sequence[V]], list[V]]"; expected "Callable[[str], list[V]]"  [arg-type]

anyio (https://github.com/agronholm/anyio)
- src/anyio/from_thread.py:395: error: Argument 1 to "submit" of "Executor" has incompatible type "Callable[[Callable[..., Awaitable[T_Retval]], VarArg(object), DefaultNamedArg(str, 'backend'), DefaultNamedArg(dict[str, Any] | None, 'backend_options')], T_Retval]"; expected "Callable[[Callable[[], Coroutine[Any, Any, None]], str, dict[str, Any] | None], T_Retval]"  [arg-type]
+ src/anyio/from_thread.py:395: error: Argument 1 to "submit" of "Executor" has incompatible type "Callable[[Callable[..., Awaitable[T_Retval]], VarArg(object), DefaultNamedArg(str, 'backend'), DefaultNamedArg(dict[str, Any] | None, 'backend_options')], T_Retval]"; expected "Callable[[Callable[[], Coroutine[Any, Any, None]], str, dict[str, Any] | None], None]"  [arg-type]

schema_salad (https://github.com/common-workflow-language/schema_salad)
+ Warning: --new-type-inference flag is deprecated; new type inference algorithm is already enabled by default

werkzeug (https://github.com/pallets/werkzeug)
+ src/werkzeug/datastructures/auth.py:175: error: Unused "type: ignore" comment  [unused-ignore]
+ src/werkzeug/datastructures/auth.py:204: error: Unused "type: ignore" comment  [unused-ignore]

cwltool (https://github.com/common-workflow-language/cwltool)
+ Warning: --new-type-inference flag is deprecated; new type inference algorithm is already enabled by default

pip (https://github.com/pypa/pip)
+ src/pip/_internal/req/req_uninstall.py:176: error: Unused "type: ignore" comment  [unused-ignore]

urllib3 (https://github.com/urllib3/urllib3)
+ dummyserver/server.py:281: error: Unused "type: ignore" comment  [unused-ignore]

koda-validate (https://github.com/keithasaurus/koda-validate)
+ koda_validate/signature.py:384: error: Returning Any from function declared to return "_DecoratedFunc"  [no-any-return]
+ koda_validate/signature.py:388: error: Returning Any from function declared to return "_DecoratedFunc | Callable[[_DecoratedFunc], _DecoratedFunc]"  [no-any-return]

Expression (https://github.com/cognitedata/Expression)
- expression/extra/parser.py:62: error: Argument 1 to "or_else" has incompatible type "Parser[_A]"; expected "Parser[_A]"  [arg-type]
- expression/extra/parser.py:65: error: Argument 1 to "map" has incompatible type "Callable[[_A], _B]"; expected "Callable[[_A], _B]"  [arg-type]
- expression/extra/parser.py:90: error: Argument 1 to "bind" has incompatible type "Callable[[_A], Parser[_B]]"; expected "Callable[[_A], Parser[_B]]"  [arg-type]
- expression/extra/parser.py:172: error: Cannot infer type argument 1 of "pipe"  [misc]
- expression/extra/parser.py:172: error: Cannot infer type argument 2 of "pipe"  [misc]
- expression/extra/parser.py:228: error: Argument 1 to "map" has incompatible type "Callable[[tuple[Any, ...]], Any]"; expected "Callable[[_A], _B]"  [arg-type]
- expression/extra/parser.py:251: error: Cannot infer type argument 1 of "pipe"  [misc]
- expression/extra/parser.py:255: error: Argument 1 to "map" has incompatible type "Callable[[tuple[Callable[[_A], _B], _A]], _B]"; expected "Callable[[_A], _B]"  [arg-type]
- expression/extra/parser.py:273: error: Argument 1 to "lift2" has incompatible type "Callable[[_A], Callable[[Block[_A]], Block[_A]]]"; expected "Callable[[_A], Callable[[_B], _C]]"  [arg-type]
- expression/extra/parser.py:287: error: Cannot infer type argument 1 of "pipe"  [misc]
- expression/extra/parser.py:287: error: Cannot infer type argument 2 of "pipe"  [misc]
- expression/extra/parser.py:287: error: Cannot infer type argument 3 of "pipe"  [misc]
- expression/extra/parser.py:292: error: Argument 1 to "map" has incompatible type "Callable[[_A], str]"; expected "Callable[[_A], _B]"  [arg-type]
- expression/extra/parser.py:292: error: Incompatible return value type (got "str", expected "_B")  [return-value]
- expression/extra/parser.py:370: error: Argument 1 to "map" has incompatible type "Callable[[_A], Some[_A] | Nothing_[_A]]"; expected "Callable[[_A], _B]"  [arg-type]
- expression/extra/parser.py:372: error: Argument 1 to "or_else" has incompatible type "Parser[Some[_A] | Nothing_[_A]]"; expected "Parser[_A]"  [arg-type]
- expression/extra/parser.py:396: error: Cannot infer type argument 1 of "pipe"  [misc]
- expression/extra/parser.py:399: error: Argument 1 to "map" has incompatible type "Callable[[tuple[_A, Any]], _A]"; expected "Callable[[_A], _B]"  [arg-type]
- expression/extra/parser.py:421: error: Cannot infer type argument 1 of "pipe"  [misc]
- expression/extra/parser.py:424: error: Argument 1 to "map" has incompatible type "Callable[[tuple[Any, _B]], _B]"; expected "Callable[[_A], _B]"  [arg-type]
- expression/extra/parser.py:440: error: Cannot infer type argument 1 of "pipe"  [misc]
- expression/extra/parser.py:440: error: Cannot infer type argument 2 of "pipe"  [misc]
- expression/extra/parser.py:440: error: Cannot infer type argument 3 of "pipe"  [misc]
- expression/extra/parser.py:444: error: Argument 1 to "and_then" has incompatible type "Parser[Block[str]]"; expected "Parser[_B]"  [arg-type]
- expression/extra/parser.py:445: error: Argument 1 to "starmap" has incompatible type "Callable[[Some[str] | Nothing_[str], Block[str]], int]"; expected "Callable[[_A, _B], _C]"  [arg-type]
- expression/extra/parser.py:474: error: Cannot infer type argument 1 of "pipe"  [misc]
- expression/extra/parser.py:474: error: Cannot infer type argument 2 of "pipe"  [misc]
- expression/extra/parser.py:476: error: Argument 1 to "and_then" has incompatible type "Parser[Block[str]]"; expected "Parser[_B]"  [arg-type]
- expression/extra/parser.py:478: error: Argument 1 to "and_then" has incompatible type "Parser[Some[_B] | Nothing_[_B]]"; expected "Parser[_B]"  [arg-type]
- expression/extra/parser.py:481: error: Argument 1 to "ignore_then" has incompatible type "Parser[Block[str]]"; expected "Parser[_B]"  [arg-type]
- expression/extra/parser.py:485: error: Argument 1 to "starmap" has incompatible type "Callable[[tuple[Some[str] | Nothing_[str], Block[str]], Some[Block[str]] | Nothing_[Block[str]]], float]"; expected "Callable[[_A, _B], _C]"  [arg-type]
- expression/extra/parser.py:512: error: Cannot infer type argument 1 of "pipe"  [misc]
- expression/extra/parser.py:524: error: Argument 1 to "lift2" has incompatible type "Callable[[str], Callable[[str], bool]]"; expected "Callable[[_A], Callable[[_B], _C]]"  [arg-type]
- expression/extra/parser.py:549: error: Argument 1 to "map" has incompatible type "Callable[[Any], None]"; expected "Callable[[_A], _B]"  [arg-type]
- tests/test_seq.py:108: error: Argument 2 to "pipe" has incompatible type "Callable[[Iterable[_TSource]], _TSource]"; expected "Callable[[list[int]], _TSource]"  [arg-type]
- tests/test_seq.py:219: error: Argument 2 to "pipe" has incompatible type "Callable[[Iterable[_TSource]], Iterable[_TSource]]"; expected "Callable[[list[int]], Iterable[_TSource]]"  [arg-type]
- tests/test_seq.py:316: error: Argument 2 to "pipe" has incompatible type "Callable[[Iterable[_TResult]], Iterable[tuple[int, _TResult]]]"; expected "Callable[[list[int]], Iterable[tuple[int, _TResult]]]"  [arg-type]
- tests/test_parser.py:223: error: Cannot infer type argument 1 of "pipe"  [misc]
+ tests/test_parser.py:223: error: Need type annotation for "rest"  [var-annotated]
+ tests/test_parser.py:225: error: Argument 2 to "pipe" has incompatible type "Callable[[Parser[_A]], Parser[Block[_A]]]"; expected "Callable[[Parser[str]], Never]"  [arg-type]
+ tests/test_parser.py:226: error: Argument 3 to "pipe" has incompatible type "Callable[[Parser[_A]], Parser[Some[_A] | Nothing_[_A]]]"; expected "Callable[[Never], Never]"  [arg-type]
- tests/test_map.py:49: error: Argument 2 to "pipe" has incompatible type "Callable[[Map[Key, Value]], Iterable[tuple[Key, Value]]]"; expected "Callable[[Map[str, int]], Iterable[tuple[Key, Value]]]"  [arg-type]
- tests/test_compose.py:68: error: Cannot infer type argument 2 of "compose"  [misc]
- tests/test_compose.py:77: error: Cannot infer type argument 1 of "compose"  [misc]