[WIP] Try native parser by ilevkivskyi · Pull Request #21331 · python/mypy

ilevkivskyi · 2026-04-25T15:15:06Z

No description provided.

ilevkivskyi · 2026-04-25T17:26:26Z

OK, so most notable things from mypy_primer are:

There is some line number churn because types in annotations now have more precise line numbers, IMO this is good.
Type comments like this don't work anymore:
```
d = [

]  # type: list[int]
```
I think we can support specifically line and end_line. IMO looking for a type comment everywhere in between in silly.

It looks like something is different about left/right associativity in the new parser. For example:

# mypy: enable-error-code="redundant-expr"
x: int
y: int
z: int
if (
    x is None  # Old parser error and new parser error
    or y is None  # Old parser error only
    or z is None
):
    pass

Not sure if need to do anything about it.

New parser has different rules about whether (redundant?) parenthesis are part of the expression, example:
```
foo = (  # Old parser error here
    "no way"  # New parser error here
) + 1
```
Again, not sure if need to do anything about this.

In any case, may be worth mentioning in release notes that some line number changes may happen. cc @JukkaL

Also some of the test failures look suspicious, most notably few mypyc test cases. @JukkaL could you please check these test cases? In the meantime I can look at the type comment issue, likely a trivial fix.

Fix is trivial, see python/mypy#21331 for context.

ilevkivskyi · 2026-04-27T15:14:48Z

OK, there is one more problem, this test fails with the new parser

[case testIgnoreInPlatformSpecificCodeUsed]
# flags: --warn-unused-ignores

import sys
assert sys.platform == "win32"

42 + "no way"  # type: ignore[operator]
[builtins fixtures/isinstancelist.pyi]

while passes with old parser. I think this is important to fix before the release.

JukkaL · 2026-04-27T15:49:25Z

I think this is important to fix before the release.

I can have a look at this.

ilevkivskyi · 2026-04-27T15:51:45Z

I can have a look at this.

OK, go ahead, I briefly looked at this, the problem is that we should update ser.skipped_lines here https://github.com/mypyc/ast_serialize/blob/main/src/serialize_ast.rs#L642

ilevkivskyi · 2026-04-27T17:39:19Z

@JukkaL did you already start working on this? Because I already have a fix, sorry.

ilevkivskyi · 2026-04-27T17:49:36Z

@JukkaL here is the fix mypyc/ast_serialize#60, please feel free to amend/merge/release it as needed.

See python/mypy#21331 for context.

This fixes some issues discovered in #21331 Two things here: * Old parser keeps `metaclass` in keywords, so it is better to keep it in the new one as well * Old parser has right-associativity for BoolOps, so should have the new one If we want to have the new behavior for some reason, it is possible to instead make some fixes "downstream" in semantic analysis/type-checking/etc. _But_, it is non-trivial, and IMO it is not a good time to change the behavior while we still have both parsers. Note I don't add test for the metaclass change because a whole bunch of mypyc tests fails with new parser already without this PR.

For some reason Ruff parser has a "bolted-on" support for debug f-strings like `f"test {some.expr()=}"`. I essentially replicate the the post-processing logic in Python parser, except for implicit collapsing of consecutive literal elements, which is (already) done by the caller. This fixes another few issues fond in python/mypy#21331

ilevkivskyi · 2026-04-30T10:40:03Z

OK, there is yet another problem with native parser affecting mypyc. Native parser sets import flags (like is_top_level, is_mypy_only) only in tree.imports but not in tree.defs. At the same time there are many places where mypyc needs those flags.

I don't see how to restore those reliably, so I guess we will need to bump cache_version again...

ilevkivskyi · 2026-04-30T10:41:32Z

I am going to proceed with ast_serializev0.2.3 release to get the f-string fixes, and will bump the cache version in v0.3.0.

Noticed this while working on the other fixes from #21331. If file doesn't exist, it will be parsed with the old parser, even if the native parser is enabled, so we need to run semantic analysis pass one on it. Not adding a test because this will not be needed soon (and I am lazy, sorry).

github-actions · 2026-04-30T12:59:27Z

Diff from mypy_primer, showing the effect of this PR on open source code:

prefect (https://github.com/PrefectHQ/prefect)
- src/prefect/utilities/collections.py:504: error: "None" object is not iterable  [misc]
+ src/prefect/utilities/collections.py:505: error: "None" object is not iterable  [misc]
- src/prefect/_internal/concurrency/api.py:181: error: "Call" expects no type arguments, but 1 given  [type-arg]
- src/prefect/_internal/concurrency/api.py:181: error: Variable "prefect._internal.concurrency.api.T" is not valid as a type  [valid-type]
+ src/prefect/_internal/concurrency/api.py:182: error: Variable "prefect._internal.concurrency.api.T" is not valid as a type  [valid-type]
- src/prefect/_internal/concurrency/api.py:181: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
+ src/prefect/_internal/concurrency/api.py:182: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
+ src/prefect/_internal/concurrency/api.py:183: error: "Call" expects no type arguments, but 1 given  [type-arg]
+ src/prefect/_internal/concurrency/api.py:183: error: Variable "prefect._internal.concurrency.api.T" is not valid as a type  [valid-type]
+ src/prefect/_internal/concurrency/api.py:183: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
- src/prefect/server/api/deployments.py:459: error: Invalid "type: ignore" comment  [syntax]
+ src/prefect/server/api/deployments.py:459: error: Unused "type: ignore" comment  [unused-ignore]
- src/prefect/workers/base.py:30: error: Invalid "type: ignore" comment  [syntax]
+ src/prefect/workers/base.py:30: error: Unused "type: ignore" comment  [unused-ignore]

operator (https://github.com/canonical/operator)
- ops/charm.py:137: error: Invalid "type: ignore" comment  [syntax]
- ops/charm.py:137: error: Incompatible types in assignment (expression has type "str | None", variable has type "str")  [assignment]

colour (https://github.com/colour-science/colour)
- colour/characterisation/aces_it.py:975: error: Ellipses cannot accompany other parameter types in function type signature  [syntax]
+ colour/characterisation/aces_it.py:976: error: Unexpected "..."  [misc]
+ colour/characterisation/aces_it.py:977: error: Unexpected "..."  [misc]

Expression (https://github.com/cognitedata/Expression)
- tests/test_fn.py:13: error: Invalid "type: ignore" comment  [syntax]
+ tests/test_fn.py:13: error: Unused "type: ignore" comment  [unused-ignore]
- tests/test_catch.py:100: error: Invalid "type: ignore" comment  [syntax]
- tests/test_catch.py:100: error: Incompatible return value type (got "int", expected "str")  [return-value]
- tests/test_catch.py:100: error: Unsupported operand types for + ("int" and "str")  [operator]

antidote (https://github.com/Finistere/antidote)
- src/antidote/lib/interface_ext/__init__.py:1359: error: "staticmethod" expects 2 type arguments, but 1 given  [type-arg]
+ src/antidote/lib/interface_ext/__init__.py:1361: error: "staticmethod" expects 2 type arguments, but 1 given  [type-arg]
- src/antidote/lib/interface_ext/__init__.py:1359: error: Missing type arguments for generic type "staticmethod"  [type-arg]
+ src/antidote/lib/interface_ext/__init__.py:1361: error: Missing type arguments for generic type "staticmethod"  [type-arg]

core (https://github.com/home-assistant/core)
- homeassistant/components/zwave_js/api.py:146: error: Parameter 1 of Literal[...] cannot be of type "Any"  [valid-type]
+ homeassistant/components/zwave_js/api.py:148: error: Parameter 1 of Literal[...] cannot be of type "Any"  [valid-type]

Tanjun (https://github.com/FasterSpeeding/Tanjun)
- tanjun/commands/menu.py:420: error: Type argument "_UserCallbackSigT" of "MessageCommand" must be a subtype of "def (MessageContext, /, *Any, **Any) -> Coroutine[Any, Any, None]"  [type-var]
+ tanjun/commands/menu.py:422: error: Type argument "_UserCallbackSigT" of "MessageCommand" must be a subtype of "def (MessageContext, /, *Any, **Any) -> Coroutine[Any, Any, None]"  [type-var]
- tanjun/commands/menu.py:420: error: Type argument "_UserCallbackSigT" of "SlashCommand" must be a subtype of "def (SlashContext, /, *Any, **Any) -> Coroutine[Any, Any, None]"  [type-var]
+ tanjun/commands/menu.py:422: error: Type argument "_UserCallbackSigT" of "SlashCommand" must be a subtype of "def (SlashContext, /, *Any, **Any) -> Coroutine[Any, Any, None]"  [type-var]
- tanjun/commands/menu.py:420: error: Type argument "_MessageCallbackSigT" of "MessageCommand" must be a subtype of "def (MessageContext, /, *Any, **Any) -> Coroutine[Any, Any, None]"  [type-var]
+ tanjun/commands/menu.py:424: error: Type argument "_MessageCallbackSigT" of "MessageCommand" must be a subtype of "def (MessageContext, /, *Any, **Any) -> Coroutine[Any, Any, None]"  [type-var]
- tanjun/commands/menu.py:420: error: Type argument "_MessageCallbackSigT" of "SlashCommand" must be a subtype of "def (SlashContext, /, *Any, **Any) -> Coroutine[Any, Any, None]"  [type-var]
+ tanjun/commands/menu.py:424: error: Type argument "_MessageCallbackSigT" of "SlashCommand" must be a subtype of "def (SlashContext, /, *Any, **Any) -> Coroutine[Any, Any, None]"  [type-var]

ibis (https://github.com/ibis-project/ibis)
- ibis/selectors.py:434: error: Variable "ibis.common.collections.frozendict" is not valid as a type  [valid-type]
+ ibis/selectors.py:437: error: Variable "ibis.common.collections.frozendict" is not valid as a type  [valid-type]
- ibis/selectors.py:434: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
+ ibis/selectors.py:437: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
- ibis/expr/types/generic.py:152: error: Function "ibis.expr.types.generic.Value.type" is not valid as a type  [valid-type]
+ ibis/expr/types/generic.py:164: error: Function "ibis.expr.types.generic.Value.type" is not valid as a type  [valid-type]
- ibis/expr/types/generic.py:152: note: Perhaps you need "Callable[...]" or a callback protocol?
+ ibis/expr/types/generic.py:164: note: Perhaps you need "Callable[...]" or a callback protocol?
- ibis/expr/types/generic.py:170: error: Function "ibis.expr.types.generic.Value.type" is not valid as a type  [valid-type]
+ ibis/expr/types/generic.py:171: error: Function "ibis.expr.types.generic.Value.type" is not valid as a type  [valid-type]
- ibis/expr/types/generic.py:170: note: Perhaps you need "Callable[...]" or a callback protocol?
+ ibis/expr/types/generic.py:171: note: Perhaps you need "Callable[...]" or a callback protocol?
- ibis/expr/types/temporal.py:428: error: "Value" expects no type arguments, but 1 given  [type-arg]
+ ibis/expr/types/temporal.py:431: error: "Value" expects no type arguments, but 1 given  [type-arg]
- ibis/expr/types/temporal.py:1161: error: "Value" expects no type arguments, but 1 given  [type-arg]
+ ibis/expr/types/temporal.py:1174: error: "Value" expects no type arguments, but 1 given  [type-arg]

static-frame (https://github.com/static-frame/static-frame)
+ static_frame/core/frame.py:7140: error: Unused "type: ignore" comment  [unused-ignore]
+ static_frame/core/frame.py:7141: error: Need type annotation for "name"  [var-annotated]
+ static_frame/core/frame.py:7383: error: Unused "type: ignore" comment  [unused-ignore]
+ static_frame/core/frame.py:7384: error: Need type annotation for "name"  [var-annotated]

apprise (https://github.com/caronc/apprise)
- apprise/plugins/fluxer.py:512: error: Unsupported operand types for + ("str" and "bool")  [operator]
+ apprise/plugins/fluxer.py:513: error: Unsupported operand types for + ("str" and "bool")  [operator]
- apprise/plugins/fluxer.py:512: note: Right operand is of type "bool | str"
+ apprise/plugins/fluxer.py:513: note: Right operand is of type "bool | str"
- apprise/plugins/fluxer.py:512: error: Incompatible types in assignment (expression has type "Any | str", target has type "bool")  [assignment]
+ apprise/plugins/fluxer.py:513: error: Incompatible types in assignment (expression has type "Any | str", target has type "bool")  [assignment]

This fixes some issues discovered in #21331 Two things here: * Old parser keeps `metaclass` in keywords, so it is better to keep it in the new one as well * Old parser has right-associativity for BoolOps, so should have the new one If we want to have the new behavior for some reason, it is possible to instead make some fixes "downstream" in semantic analysis/type-checking/etc. _But_, it is non-trivial, and IMO it is not a good time to change the behavior while we still have both parsers. Note I don't add test for the metaclass change because a whole bunch of mypyc tests fails with new parser already without this PR.

ilevkivskyi · 2026-04-30T13:02:24Z

OK, I am happy with the primer now.

Noticed this while working on the other fixes from #21331. If file doesn't exist, it will be parsed with the old parser, even if the native parser is enabled, so we need to run semantic analysis pass one on it. Not adding a test because this will not be needed soon (and I am lazy, sorry).

[WIP] Try native parser

e5fca3e

ilevkivskyi marked this pull request as draft April 25, 2026 15:15

This comment has been minimized.

Sign in to view

ilevkivskyi mentioned this pull request Apr 25, 2026

Support type comment on the last line of an assignment mypyc/ast_serialize#57

Merged

ilevkivskyi added a commit to mypyc/ast_serialize that referenced this pull request Apr 27, 2026

Support type comment on the last line of an assignment (#57)

1f78fcf

Fix is trivial, see python/mypy#21331 for context.

ilevkivskyi mentioned this pull request Apr 27, 2026

Mark statements after top-level false assert as skipped mypyc/ast_serialize#60

Merged

ilevkivskyi added a commit to mypyc/ast_serialize that referenced this pull request Apr 29, 2026

Mark statements after top-level false assert as skipped (#60)

53e49f6

See python/mypy#21331 for context.

Merge remote-tracking branch 'upstream/master' into try-native-parser

e875d15

This comment has been minimized.

Sign in to view

This was referenced Apr 29, 2026

Make new parser consistent with the old one #21377

Merged

Support debug f-string elements mypyc/ast_serialize#61

Merged

ilevkivskyi mentioned this pull request Apr 30, 2026

Run semantic analysis pass one if file doesn't exsit #21379

Merged

Merge remote-tracking branch 'upstream/master' into try-native-parser

b65f65f

ilevkivskyi closed this Apr 30, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[WIP] Try native parser#21331

[WIP] Try native parser#21331
ilevkivskyi wants to merge 3 commits intopython:masterfrom
ilevkivskyi:try-native-parser

ilevkivskyi commented Apr 25, 2026

Uh oh!

This comment has been minimized.

ilevkivskyi commented Apr 25, 2026

Uh oh!

ilevkivskyi commented Apr 27, 2026

Uh oh!

JukkaL commented Apr 27, 2026

Uh oh!

ilevkivskyi commented Apr 27, 2026

Uh oh!

ilevkivskyi commented Apr 27, 2026

Uh oh!

ilevkivskyi commented Apr 27, 2026

Uh oh!

This comment has been minimized.

ilevkivskyi commented Apr 30, 2026

Uh oh!

ilevkivskyi commented Apr 30, 2026

Uh oh!

github-actions Bot commented Apr 30, 2026

Uh oh!

ilevkivskyi commented Apr 30, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

ilevkivskyi commented Apr 25, 2026

Uh oh!

This comment has been minimized.

ilevkivskyi commented Apr 25, 2026

Uh oh!

ilevkivskyi commented Apr 27, 2026

Uh oh!

JukkaL commented Apr 27, 2026

Uh oh!

ilevkivskyi commented Apr 27, 2026

Uh oh!

ilevkivskyi commented Apr 27, 2026

Uh oh!

ilevkivskyi commented Apr 27, 2026

Uh oh!

This comment has been minimized.

ilevkivskyi commented Apr 30, 2026

Uh oh!

ilevkivskyi commented Apr 30, 2026

Uh oh!

github-actions Bot commented Apr 30, 2026

Uh oh!

ilevkivskyi commented Apr 30, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants