ast.Call end_lineno is defined and returns None

[gaborbernat]

BPO	40726
Nosy	@serhiy-storchaka, @asottile, @pablogsal, @miss-islington, @remilapeyre, @gaborbernat, @isidentical, @laloch
PRs	bpo-40726: handle uninitalized end_lineno on ast.increment_lineno #20312 [3.9] bpo-40726: handle uninitalized end_lineno on ast.increment_lineno (GH-20312) #21741 [3.8] bpo-40726: handle uninitalized end_lineno on ast.increment_line… #21742 [3.8] bpo-40726: handle uninitalized end_lineno on ast.increment_lineno #21745

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2020-08-05.19:34:48.912>
created_at = <Date 2020-05-22.08:34:59.646>
labels = ['type-bug', 'library', '3.9', '3.10']
title = 'ast.Call end_lineno is defined and returns None'
updated_at = <Date 2020-08-05.19:34:48.911>
user = 'https://github.com/gaborbernat'

bugs.python.org fields:

activity = <Date 2020-08-05.19:34:48.911>
actor = 'pablogsal'
assignee = 'none'
closed = True
closed_date = <Date 2020-08-05.19:34:48.912>
closer = 'pablogsal'
components = ['Library (Lib)']
creation = <Date 2020-05-22.08:34:59.646>
creator = 'gaborjbernat'
dependencies = []
files = []
hgrepos = []
issue_num = 40726
keywords = ['patch']
message_count = 13.0
messages = ['369567', '369569', '369572', '369574', '369577', '369581', '369614', '369619', '369621', '369623', '374880', '374882', '374893']
nosy_count = 8.0
nosy_names = ['serhiy.storchaka', 'Anthony Sottile', 'pablogsal', 'miss-islington', 'remi.lapeyre', 'gaborjbernat', 'BTaskaya', 'laloch']
pr_nums = ['20312', '21741', '21742', '21745']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue40726'
versions = ['Python 3.9', 'Python 3.10']

[gaborbernat]

Reporting an issue from xonsh/xonsh#3581; boils down to, ast.Call used to not define end_lineno in 3.8:

py -3.8 -c 'import ast; type(ast.Call().end_lineno)'                                                                                                                                                                 Traceback (most recent call last):
  File "<string>", line 1, in <module>
AttributeError: 'Call' object has no attribute 'end_lineno'

However in 3.9 is defined with None:

py -3.9 -c 'import ast; type(ast.Call().end_lineno)'

This messes with some other operations in ast, namely https://github.com/python/cpython/blob/master/Lib/ast.py#L233

[laloch]

The issue is not limited to ast.Call. Other AST nodes are also affected (e.g. ast.BinOp). It's also not limited to end_lineno attribute. The same applies to end_col_offset.

[isidentical]

This is because the 'end_lineno' and 'end_col_offset' are declared as optional attributes in the ASDL spec. The commit 'b7e9525f9c7ef02a1d2ad8253afdeb733b0951d4' made all optional fields and attributes auto initalized with None.

[isidentical]

See here for the complete spec: https://docs.python.org/3.9/library/ast.html#abstract-grammar

[remilapeyre]

This was done in b7e9525 for bpo-36287, all attribute should now be defined, even when they are not set. It looks like some parts of the ast module where not updated.

If nobody works on this I will send a PR to update the rest of AST later today.

[isidentical]

If nobody works on this I will send a PR to update the rest of AST later today.

:/ I'm terribly sorry for that, I just sent a PR without refreshing the tab.

[asottile]

There's current expectation in a lot of linters / code formatters that the *lineno and *col_offset attributes will be missing if they are not attached to the node

setting them to None is going to break a lot of those, if possible I'd suggest going back to when they were missing

[isidentical]

setting them to None is going to break a lot of those, if possible I'd suggest going back to when they were missing

'lineno' and 'col_offset' will never be none, since both are declared as normal integers. But on the other hand, 'end_lineno' and 'end_col_offset' are declared as optional integers which would make sense to auto initalize them with *None*, and I dont think it would ever break some code that complies with the ASDL declaration.

[laloch]

Actually, Xonsh (http://github.com/xonsh/xonsh) tests show that keyword AST nodes are missing 'lineno' attribute, but that could be our fault.

[laloch]

Actually, Xonsh tests show that keyword AST nodes are missing 'lineno' attribute, but that could be our fault.

Yes, our fault. Sorry for the noise.

[pablogsal]

New changeset 8f4380d by Batuhan Taskaya in branch 'master':
bpo-40726: handle uninitalized end_lineno on ast.increment_lineno (GH-20312)
8f4380d

[miss-islington]

New changeset a132098 by Miss Islington (bot) in branch '3.9':
bpo-40726: handle uninitalized end_lineno on ast.increment_lineno (GH-20312)
a132098

[miss-islington]

New changeset b24c9d2 by Batuhan Taskaya in branch '3.8':
[3.8] bpo-40726: handle uninitalized end_lineno on ast.increment_lineno (GH-21745)
b24c9d2