Carry comments with the AST

[brettcannon]

BPO	24119
Nosy	@gvanrossum, @brettcannon, @rhettinger, @pitrou, @tiran, @markshannon, @serhiy-storchaka, @kernc, @mlouielu, @mbdevpl, @isidentical

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2015-05-03.15:21:10.716>
labels = ['type-feature', 'library']
title = 'Carry comments with the AST'
updated_at = <Date 2020-03-13.01:04:47.072>
user = 'https://github.com/brettcannon'

bugs.python.org fields:

activity = <Date 2020-03-13.01:04:47.072>
actor = 'gvanrossum'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2015-05-03.15:21:10.716>
creator = 'brett.cannon'
dependencies = []
files = []
hgrepos = []
issue_num = 24119
keywords = []
message_count = 15.0
messages = ['242485', '242499', '242560', '242602', '242603', '289643', '289685', '292655', '292661', '292682', '292683', '333540', '333541', '334072', '364061']
nosy_count = 11.0
nosy_names = ['gvanrossum', 'brett.cannon', 'rhettinger', 'pitrou', 'christian.heimes', 'Mark.Shannon', 'serhiy.storchaka', 'kernc', 'louielu', 'mbdevpl', 'BTaskaya']
pr_nums = []
priority = 'low'
resolution = None
stage = 'test needed'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue24119'
versions = ['Python 3.5']

[brettcannon]

One thing about https://www.python.org/dev/peps/pep-0484/ is that it makes comments potentially semantically meaningful. Unfortunately the AST doesn't carry comments with it in any way, making it difficult to build a tool to implement a linter for PEP-484 using purely the ast module. Even if comments were carried along side-band and could do correlation by line number would be useful in this scenario.

I thought an issue had previously existed for this topic but I could find it.

[markshannon]

Comments don't belong on the AST. Where would you attach them?

The tokenizer module provides all information about comments. Tools can get the information quite easily if they need it.

[brettcannon]

Normally I would agree comments don't belong there, but if we are going to start giving them semantic meaning then I don't think it's not so clear to me anymore.

As to where to attach, simple place is off of the Module node. Another is to have it be fundamental like lineno and only attach it when it is a line-trailing comment.

Yes, the tokenize module will give you the comments as well, but it is unfortunate you have to parse the code twice in order to get the comments and the AST.

[brettcannon]

Another option is to provide a tool in 'tokenize' or 'ast' which will take the source and some comment regex and then attach the found comment metadata to the AST.

[pitrou]

Or a separate AST node -> comment mapping.

[mbdevpl]

For some time now, there's an alternate ast implementation https://github.com/python/typed_ast that carries PEP-484 type comments with the AST as attributes of certain nodes.

Their approach is described here: https://github.com/python/typed_ast/blob/master/typed_ast/ast3.py#L5

If type comments become mainstream in Python, could this approach maybe be adopted as official Python AST at some point?

[brettcannon]

The type annotation is already in the AST so there's nothing to carry over from typed_ast (we only care about the latest Python version while typed_ast tries to be version-agnostic).

[mlouielu]

Brett, which implement method will you prefer?

If we want to carry comment at builtin_compile_impl, it will need to change the grammar since tokenize just drop the comment when dealing with source code.

But if just using regex, will it be more easy with just combine the exists tokenize and ast module?

[rhettinger]

After PEP-526, the need for this proposal may have evaporated.

[brettcannon]

There's potentially some usefulness from other tools, but Raymond is right that the main motivation is definitely gone long-term. Dropping this down to "low" priority simply because others have asked for this kind of support before.

[serhiy-storchaka]

Where would comment be attached in the following case?

a = ('start'         # comment 1
     'continuation') # comment 2

[rhettinger]

Since the AST carries the module/lineno attributes isn't there already a way trace back into the token stream to recover comments?

[rhettinger]

Possible superceder: https://bugs.python.org/issue33337

[gvanrossum]

See also bpo-35766.

[gvanrossum]

I propose to close this issue, since (as of Python 3.8) we now have ast.parse(source, type_comments=True).