ast col_offset wrong for list comprehensions, generators and tuples

[samuelcolvin]

BPO	31241
Nosy	@brettcannon, @benjaminp, @serhiy-storchaka, @1st1, @samuelcolvin
PRs	bpo-31241: Fix AST node position for list and generator comprehensions. #10633

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/serhiy-storchaka'
closed_at = <Date 2018-11-27.07:42:56.775>
created_at = <Date 2017-08-20.19:03:53.905>
labels = ['interpreter-core', '3.8']
title = 'ast col_offset wrong for list comprehensions, generators and tuples'
updated_at = <Date 2018-11-27.07:42:56.773>
user = 'https://github.com/samuelcolvin'

bugs.python.org fields:

activity = <Date 2018-11-27.07:42:56.773>
actor = 'serhiy.storchaka'
assignee = 'serhiy.storchaka'
closed = True
closed_date = <Date 2018-11-27.07:42:56.775>
closer = 'serhiy.storchaka'
components = ['Interpreter Core']
creation = <Date 2017-08-20.19:03:53.905>
creator = 'samuelcolvin'
dependencies = []
files = []
hgrepos = []
issue_num = 31241
keywords = ['patch']
message_count = 5.0
messages = ['300606', '330194', '330385', '330470', '330497']
nosy_count = 5.0
nosy_names = ['brett.cannon', 'benjamin.peterson', 'serhiy.storchaka', 'yselivanov', 'samuelcolvin']
pr_nums = ['10633']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue31241'
versions = ['Python 3.8']

[samuelcolvin]

With Python 3.5 and 3.6 list comprehensions, generators and tuples have the col_offset for their ast nodes off by 1:

import ast
ast.parse('{a for a in range(3)}').body[0].value.col_offset
>> 0  # set comprehension correct

ast.parse('{a: 1 for a in range(3)}').body[0].value.col_offset
>> 0  # dict comprehension correct

ast.parse('[a for a in range(3)]').body[0].value.col_offset
>> 1  # list comprehension wrong!

ast.parse('(a for a in range(3))').body[0].value.col_offset
>> 1  # generator comprehension wrong!

ast.parse('[1, 2, 3]').body[0].value.col_offset
>> 0  # list correct

ast.parse('{1, 2, 3}').body[0].value.col_offset
>> 0  # set correct

ast.parse('{1: 1, 2: 2, 3: 3}').body[0].value.col_offset
>> 0  # dict correct

ast.parse('(1, 2, 3)').body[0].value.col_offset
>> 1  # tuple wrong!

I haven't tried 3.4, the issue could be there too.

There are some other related issues bpo-16806 and bpo-21295 but they don't seem quite the same.

[serhiy-storchaka]

For list comprehensions and generator expressions this is definitely a bug. But tuples syntax technically does not include surrounded parentheses.

There is also a problem with generator expression passes as a single argument. Generator expression parentheses can be collapsed with function call parentheses: f(a for a in b).

PR 10633 makes the following changes:

• Fixes position for list comprehensions and generator expressions.

• If generator expression parentheses are be collapsed with function call parentheses, the position of the AST node for the generator expression points to the left parenthesis.

• For tuples surrounded with parentheses, the position of the AST node points to the left brace. For tuples without parentheses, it points to the position of the first tuple item.

I am not sure whether these changes should be backported to maintained versions.

[serhiy-storchaka]

I am not sure what parts of this PR should be backported if either.

[brettcannon]

Yeah, it's a tough call because it's one of those things others have probably worked around already, so backporting would break the work-arounds.

If you don't want to bother backporting, Serhiy, I think it would be fine to not do it.

[serhiy-storchaka]

New changeset b619b09 by Serhiy Storchaka in branch 'master':
bpo-31241: Fix AST node position for list and generator comprehensions. (GH-10633)
b619b09