Line number of docstring in AST

[myint]

BPO	30497
Nosy	@vstinner, @methane, @serhiy-storchaka

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2017-05-29.17:41:19.169>
created_at = <Date 2017-05-28.18:44:26.274>
labels = ['interpreter-core', '3.7']
title = 'Line number of docstring in AST'
updated_at = <Date 2017-05-29.17:59:27.288>
user = 'https://bugs.python.org/myint'

bugs.python.org fields:

activity = <Date 2017-05-29.17:59:27.288>
actor = 'myint'
assignee = 'none'
closed = True
closed_date = <Date 2017-05-29.17:41:19.169>
closer = 'serhiy.storchaka'
components = ['Interpreter Core']
creation = <Date 2017-05-28.18:44:26.274>
creator = 'myint'
dependencies = []
files = []
hgrepos = []
issue_num = 30497
keywords = []
message_count = 5.0
messages = ['294654', '294673', '294688', '294706', '294708']
nosy_count = 4.0
nosy_names = ['vstinner', 'methane', 'serhiy.storchaka', 'myint']
pr_nums = []
priority = 'normal'
resolution = 'wont fix'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue30497'
versions = ['Python 3.7']

[myint]

Since bpo-29463, it is no longer obvious how to get the line number of a docstring in the AST:

    import ast

    x = ast.parse('''\
    def foo():
        """This is a docstring."""
    ''')

    # In Python 3.6, the docstring and line number would be:
    print(x.body[0].body[0].value.s)
    print(x.body[0].body[0].value.lineno)

    # In Python 3.7, I don't know what the equivalent would be.
    print(x.body[0].docstring)
    # Line number?

We use this feature in pyflakes (PyCQA/pyflakes#271).

[serhiy-storchaka]

It is not possible anymore. Why you need the line number of a docstring?

The closest approximation is between node.lineno and node.body[0].lineno (if node.body is not empty).

[vstinner]

We use this feature in pyflakes (PyCQA/pyflakes#271)

AST is not designed to be 1-to-1 .py source to AST mapping. If you need the exact line number, you might use your own parser. I don't know what is using pylint for example? There is also https://github.com/PyCQA/redbaron which provides such 1-to-1 mapping but it has a different API than AST.

Sadly, I suggest to close this issue as WONTFIX.

[serhiy-storchaka]

Even if the line number of a docstring is known, it is not easy to determine the line number corresponding to the particular line in a docstring if it contains backslashes following by newline.

'''foo
bar
'''

and

'''\
foo
bar
'''

are equal strings, but the lines 'bar' have different offsets from the start of the strings.

The only robust method was parsing the source code starting from the start of a docstring. Now you just need to start parsing from the start of the function/class.

[myint]

I think what you guys have brought up makes sense.

Now that you mention it, I see that pyflakes gives the wrong line number if an escaped newline appears after the doctests. Though, it works fine if the escaped newline appears before it.

https://github.com/PyCQA/pyflakes/blob/1af4f14ad4675bf5c61c47bbb7c2421b50d1cba4/pyflakes/checker.py#L842

This is a non-default feature in pyflakes anyway, so we can live with it.

Thanks