ast.get_source_segment behaviour with missing location info doesn't match docstring

[iritkatriel]

BPO	40662
Nosy	@brettcannon, @benjaminp, @1st1, @pablogsal, @isidentical, @hauntsaninja, @iritkatriel
PRs	bpo-40662: fix ast.get_source_segment on node with missing location info #20155 bpo-40662: add test for ast.get_source_segment with missing line/col… #20157

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2020-05-18.18:17:15.150>
created_at = <Date 2020-05-17.17:35:18.755>
labels = ['type-bug', 'library', '3.9']
title = "ast.get_source_segment behaviour with missing location info doesn't match docstring"
updated_at = <Date 2020-05-18.18:17:15.150>
user = 'https://github.com/iritkatriel'

bugs.python.org fields:

activity = <Date 2020-05-18.18:17:15.150>
actor = 'pablogsal'
assignee = 'none'
closed = True
closed_date = <Date 2020-05-18.18:17:15.150>
closer = 'pablogsal'
components = ['Library (Lib)']
creation = <Date 2020-05-17.17:35:18.755>
creator = 'iritkatriel'
dependencies = []
files = []
hgrepos = []
issue_num = 40662
keywords = ['3.9regression']
message_count = 3.0
messages = ['369139', '369155', '369282']
nosy_count = 7.0
nosy_names = ['brett.cannon', 'benjamin.peterson', 'yselivanov', 'pablogsal', 'BTaskaya', 'hauntsaninja', 'iritkatriel']
pr_nums = ['20155', '20157']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue40662'
versions = ['Python 3.9']

[iritkatriel]

The doctoring says "If some location information (lineno, end_lineno, col_offset, or end_col_offset) is missing, return None."

However:

>>> s = "12"
>>> node = ast.parse(s).body[0]
>>> ast.get_source_segment(s, node)
'12'
>>> del node.lineno
>>> ast.get_source_segment(s, node)
>>> node = ast.parse(s).body[0]
>>> del node.end_lineno
>>> ast.get_source_segment(s, node)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/iritkatriel/src/cpython/Lib/ast.py", line 336, in get_source_segment
    end_lineno = node.end_lineno - 1
TypeError: unsupported operand type(s) for -: 'NoneType' and 'int'

In other parts of ast.py there are hasattr checks for lineno/col_offset alongside checks for None for end_lineno/end_col_offset. So this difference in behaviour of the end_ fields seems intended.

[hauntsaninja]

The code works on 3.8 for me, but has regressed on 3.9 master. Looks like this is caused by https://bugs.python.org/issue36287 (#18843)

[pablogsal]

New changeset e6578a2 by Irit Katriel in branch 'master':
bpo-40662: Fixed ast.get_source_segment for ast nodes that have incomplete location information (GH-20157)
e6578a2