Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Improve the constructors of AST nodes #105858

Closed
JelleZijlstra opened this issue Jun 16, 2023 · 6 comments
Closed

Improve the constructors of AST nodes #105858

JelleZijlstra opened this issue Jun 16, 2023 · 6 comments
Labels
stdlib Python modules in the Lib dir topic-parser type-feature A feature request or enhancement

Comments

@JelleZijlstra
Copy link
Member

JelleZijlstra commented Jun 16, 2023 •
edited by bedevere-app bot

Currently, the constructors for AST nodes accept arbitrary keyword arguments and don't enforce any value:

>>> node=ast.FunctionDef(what="is this")
>>> node.what
'is this'
>>> node.name
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'FunctionDef' object has no attribute 'name'

Problems with the current situation:

  • To make a correct AST node, you have to pass every attribute, including ones that could sensibly default to an empty list
  • You cannot rely on attributes like name being present
  • It's possible to create useless, broken AST nodes like the above
  • Introspection tools can't easily tell what the signature of the constructor is supposed to be
  • Adding new fields to an AST node causes various compatibility issues (PEP-695: Potentially breaking changes made to __match_args__ attributes of AST nodes #104799)

Proposed solution for 3.13:

  • Default all fields to some sensible value if they are not provided to the constructor: an empty list for list fields, and None for other fields.
  • Emit a DeprecationWarning if the constructor is passed arguments that it doesn't recognize (e.g., what above). In 3.15, this will raise an error.
  • Add a __text_signature__ to the AST classes indicating the expected signature.

Linked PRs
@JelleZijlstra JelleZijlstra added type-feature A feature request or enhancement topic-parser labels Jun 16, 2023
This was referenced Jun 16, 2023
Merged
Closed
@furkanonder
Copy link
Sponsor Contributor

CC: @isidentical

JelleZijlstra added a commit to JelleZijlstra/cpython that referenced this issue Jun 17, 2023
@JelleZijlstra
pythongh-105858: Improve AST node constructors
b1b73d9 
Demonstration:

>>> ast.FunctionDef.__annotations__
{'name': <class 'str'>, 'args': <class 'ast.arguments'>, 'body': list[ast.stmt], 'decorator_list': list[ast.expr], 'returns': ast.expr | None, 'type_comment': str | None, 'type_params': list[ast.type_param]}
>>> ast.FunctionDef()
<stdin>:1: DeprecationWarning: FunctionDef.__init__ missing 1 required positional argument: 'name'. This will become an error in Python 3.15.
<stdin>:1: DeprecationWarning: FunctionDef.__init__ missing 1 required positional argument: 'args'. This will become an error in Python 3.15.
<ast.FunctionDef object at 0x101959460>
>>> node = ast.FunctionDef(name="foo", args=ast.arguments())
>>> node.decorator_list
[]
>>> ast.FunctionDef(whatever="you want", name="x", args=ast.arguments())
<stdin>:1: DeprecationWarning: FunctionDef.__init__ got an unexpected keyword argument 'whatever'. Support for arbitrary keyword arguments is deprecated and will be removed in Python 3.15.
<ast.FunctionDef object at 0x1019581f0>

Known problems:
- Subclasses of AST nodes don't work properly, because we don't look up __annotations__ on the
  right class.
- Unpickling throws DeprecationWarnings, probably because of how we construct the unpickled
  object.

Need to think more about how to handle those cases.
@bedevere-bot bedevere-bot mentioned this issue Jun 17, 2023
Merged
@brandtbucher
Copy link
Member

Yes please! This has been such a pain point for me.

One note: we should probably only default optional fields (?) to None, and require everything else.

@JelleZijlstra
Copy link
Member Author

JelleZijlstra commented Jun 20, 2023 •
edited

One note: we should probably only default optional fields (?) to None, and require everything else.

Agree, I ended up doing that in my draft PR. Omitting a required field (e.g. FunctionDef.name) will raise a DeprecationWarning, as will passing a bogus field.

@iritkatriel iritkatriel added the stdlib Python modules in the Lib dir label Nov 27, 2023
JelleZijlstra added a commit that referenced this issue Feb 28, 2024
@JelleZijlstra
gh-105858: Improve AST node constructors (#105880)
ed4dfd8 
Demonstration:

>>> ast.FunctionDef.__annotations__
{'name': <class 'str'>, 'args': <class 'ast.arguments'>, 'body': list[ast.stmt], 'decorator_list': list[ast.expr], 'returns': ast.expr | None, 'type_comment': str | None, 'type_params': list[ast.type_param]}
>>> ast.FunctionDef()
<stdin>:1: DeprecationWarning: FunctionDef.__init__ missing 1 required positional argument: 'name'. This will become an error in Python 3.15.
<stdin>:1: DeprecationWarning: FunctionDef.__init__ missing 1 required positional argument: 'args'. This will become an error in Python 3.15.
<ast.FunctionDef object at 0x101959460>
>>> node = ast.FunctionDef(name="foo", args=ast.arguments())
>>> node.decorator_list
[]
>>> ast.FunctionDef(whatever="you want", name="x", args=ast.arguments())
<stdin>:1: DeprecationWarning: FunctionDef.__init__ got an unexpected keyword argument 'whatever'. Support for arbitrary keyword arguments is deprecated and will be removed in Python 3.15.
<ast.FunctionDef object at 0x1019581f0>
@JelleZijlstra
Copy link
Member Author

Merged!

Privat33r-dev pushed a commit to Privat33r-dev/cpython that referenced this issue Feb 28, 2024
@JelleZijlstra
pythongh-105858: Improve AST node constructors (python#105880)
1745629 
Demonstration:

>>> ast.FunctionDef.__annotations__
{'name': <class 'str'>, 'args': <class 'ast.arguments'>, 'body': list[ast.stmt], 'decorator_list': list[ast.expr], 'returns': ast.expr | None, 'type_comment': str | None, 'type_params': list[ast.type_param]}
>>> ast.FunctionDef()
<stdin>:1: DeprecationWarning: FunctionDef.__init__ missing 1 required positional argument: 'name'. This will become an error in Python 3.15.
<stdin>:1: DeprecationWarning: FunctionDef.__init__ missing 1 required positional argument: 'args'. This will become an error in Python 3.15.
<ast.FunctionDef object at 0x101959460>
>>> node = ast.FunctionDef(name="foo", args=ast.arguments())
>>> node.decorator_list
[]
>>> ast.FunctionDef(whatever="you want", name="x", args=ast.arguments())
<stdin>:1: DeprecationWarning: FunctionDef.__init__ got an unexpected keyword argument 'whatever'. Support for arbitrary keyword arguments is deprecated and will be removed in Python 3.15.
<ast.FunctionDef object at 0x1019581f0>
@JelleZijlstra JelleZijlstra mentioned this issue Feb 28, 2024
Closed
@JelleZijlstra
Copy link
Member Author

Reopening as I broke all the buildbots

@JelleZijlstra JelleZijlstra reopened this Feb 28, 2024
JelleZijlstra added a commit to JelleZijlstra/cpython that referenced this issue Feb 28, 2024
@JelleZijlstra
pythongh-105858: Expose some union-related objects as internal APIs
5142722 
We now use these in the AST parsing code after pythongh-105880. A few comparable types (e.g.,
NoneType) are already exposed as internal APIs.
@bedevere-app bedevere-app bot mentioned this issue Feb 28, 2024
Merged
gvanrossum pushed a commit to gvanrossum/cpython that referenced this issue Feb 28, 2024
@JelleZijlstra @gvanrossum
pythongh-105858: Improve AST node constructors (python#105880)
ab5339f 
Demonstration:

>>> ast.FunctionDef.__annotations__
{'name': <class 'str'>, 'args': <class 'ast.arguments'>, 'body': list[ast.stmt], 'decorator_list': list[ast.expr], 'returns': ast.expr | None, 'type_comment': str | None, 'type_params': list[ast.type_param]}
>>> ast.FunctionDef()
<stdin>:1: DeprecationWarning: FunctionDef.__init__ missing 1 required positional argument: 'name'. This will become an error in Python 3.15.
<stdin>:1: DeprecationWarning: FunctionDef.__init__ missing 1 required positional argument: 'args'. This will become an error in Python 3.15.
<ast.FunctionDef object at 0x101959460>
>>> node = ast.FunctionDef(name="foo", args=ast.arguments())
>>> node.decorator_list
[]
>>> ast.FunctionDef(whatever="you want", name="x", args=ast.arguments())
<stdin>:1: DeprecationWarning: FunctionDef.__init__ got an unexpected keyword argument 'whatever'. Support for arbitrary keyword arguments is deprecated and will be removed in Python 3.15.
<ast.FunctionDef object at 0x1019581f0>
encukou pushed a commit that referenced this issue Feb 28, 2024
@JelleZijlstra
gh-105858: Expose some union-related objects as internal APIs (GH-116025
d53560d 
)

We now use these in the AST parsing code after gh-105880. A few comparable types (e.g.,
NoneType) are already exposed as internal APIs.
woodruffw pushed a commit to woodruffw-forks/cpython that referenced this issue Mar 4, 2024
@JelleZijlstra @woodruffw
pythongh-105858: Improve AST node constructors (python#105880)
7ddc0d8 
Demonstration:

>>> ast.FunctionDef.__annotations__
{'name': <class 'str'>, 'args': <class 'ast.arguments'>, 'body': list[ast.stmt], 'decorator_list': list[ast.expr], 'returns': ast.expr | None, 'type_comment': str | None, 'type_params': list[ast.type_param]}
>>> ast.FunctionDef()
<stdin>:1: DeprecationWarning: FunctionDef.__init__ missing 1 required positional argument: 'name'. This will become an error in Python 3.15.
<stdin>:1: DeprecationWarning: FunctionDef.__init__ missing 1 required positional argument: 'args'. This will become an error in Python 3.15.
<ast.FunctionDef object at 0x101959460>
>>> node = ast.FunctionDef(name="foo", args=ast.arguments())
>>> node.decorator_list
[]
>>> ast.FunctionDef(whatever="you want", name="x", args=ast.arguments())
<stdin>:1: DeprecationWarning: FunctionDef.__init__ got an unexpected keyword argument 'whatever'. Support for arbitrary keyword arguments is deprecated and will be removed in Python 3.15.
<ast.FunctionDef object at 0x1019581f0>
woodruffw pushed a commit to woodruffw-forks/cpython that referenced this issue Mar 4, 2024
@JelleZijlstra @woodruffw
pythongh-105858: Expose some union-related objects as internal APIs (p…
a5fd325 
…ythonGH-116025)

We now use these in the AST parsing code after pythongh-105880. A few comparable types (e.g.,
NoneType) are already exposed as internal APIs.
adorilson pushed a commit to adorilson/cpython that referenced this issue Mar 25, 2024
@JelleZijlstra @adorilson
pythongh-105858: Improve AST node constructors (python#105880)
052e900 
Demonstration:

>>> ast.FunctionDef.__annotations__
{'name': <class 'str'>, 'args': <class 'ast.arguments'>, 'body': list[ast.stmt], 'decorator_list': list[ast.expr], 'returns': ast.expr | None, 'type_comment': str | None, 'type_params': list[ast.type_param]}
>>> ast.FunctionDef()
<stdin>:1: DeprecationWarning: FunctionDef.__init__ missing 1 required positional argument: 'name'. This will become an error in Python 3.15.
<stdin>:1: DeprecationWarning: FunctionDef.__init__ missing 1 required positional argument: 'args'. This will become an error in Python 3.15.
<ast.FunctionDef object at 0x101959460>
>>> node = ast.FunctionDef(name="foo", args=ast.arguments())
>>> node.decorator_list
[]
>>> ast.FunctionDef(whatever="you want", name="x", args=ast.arguments())
<stdin>:1: DeprecationWarning: FunctionDef.__init__ got an unexpected keyword argument 'whatever'. Support for arbitrary keyword arguments is deprecated and will be removed in Python 3.15.
<ast.FunctionDef object at 0x1019581f0>
adorilson pushed a commit to adorilson/cpython that referenced this issue Mar 25, 2024
@JelleZijlstra @adorilson
pythongh-105858: Expose some union-related objects as internal APIs (p…
7eb8096 
…ythonGH-116025)

We now use these in the AST parsing code after pythongh-105880. A few comparable types (e.g.,
NoneType) are already exposed as internal APIs.
@hroncok
Copy link
Contributor

hroncok commented Mar 26, 2024

There might be a regression: #117266

diegorusso pushed a commit to diegorusso/cpython that referenced this issue Apr 17, 2024
@JelleZijlstra @diegorusso
pythongh-105858: Improve AST node constructors (python#105880)
3a32841 
Demonstration:

>>> ast.FunctionDef.__annotations__
{'name': <class 'str'>, 'args': <class 'ast.arguments'>, 'body': list[ast.stmt], 'decorator_list': list[ast.expr], 'returns': ast.expr | None, 'type_comment': str | None, 'type_params': list[ast.type_param]}
>>> ast.FunctionDef()
<stdin>:1: DeprecationWarning: FunctionDef.__init__ missing 1 required positional argument: 'name'. This will become an error in Python 3.15.
<stdin>:1: DeprecationWarning: FunctionDef.__init__ missing 1 required positional argument: 'args'. This will become an error in Python 3.15.
<ast.FunctionDef object at 0x101959460>
>>> node = ast.FunctionDef(name="foo", args=ast.arguments())
>>> node.decorator_list
[]
>>> ast.FunctionDef(whatever="you want", name="x", args=ast.arguments())
<stdin>:1: DeprecationWarning: FunctionDef.__init__ got an unexpected keyword argument 'whatever'. Support for arbitrary keyword arguments is deprecated and will be removed in Python 3.15.
<ast.FunctionDef object at 0x1019581f0>
diegorusso pushed a commit to diegorusso/cpython that referenced this issue Apr 17, 2024
@JelleZijlstra @diegorusso
pythongh-105858: Expose some union-related objects as internal APIs (p…
e7f3eda 
…ythonGH-116025)

We now use these in the AST parsing code after pythongh-105880. A few comparable types (e.g.,
NoneType) are already exposed as internal APIs.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
stdlib Python modules in the Lib dir topic-parser type-feature A feature request or enhancement
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@JelleZijlstra @iritkatriel @hroncok @furkanonder @brandtbucher