Expose ast.unparse in the ast module

[pablogsal]

BPO	38870
Nosy	@ilevkivskyi, @pablogsal, @miss-islington, @brandtbucher, @isidentical, @CyberSaxosTiGER
PRs	bpo-38870: Expose a function to unparse an ast object in the ast module #17302 bpo-38870: Remove dependency on contextlib to avoid performance regression on import #17376 bpo-38870: Implement Simple Preceding to AST Unparser #17377 bpo-38870: refactor delimiting with context managers #17612 bpo-38870: Remove dead code related with argument unparsing #17613 Revert "bpo-38870: Remove dependency on contextlib to avoid performance regression on import (GH-17376)" #17687 bpo-38870: Run always tests that heavily use grammar features in test_unparse #17738 bpo-38870: Fix error when running with -uall in test_unparse #17739 bpo-38870: Docstring support for function/class/module nodes #17760 bpo-38870: Implement round tripping support for typed AST #17797 bpo-38870: Throw ValueError on invalid yield from usage #17798 bpo-38870: Simplify tuple like interleaves, roundtrip extslice properly #17892 bpo-38870: Correctly unparse empty docstrings #18768 bpo-38870: Implement support for ast.FunctionType #19016 bpo-38870: Don't start generated output with newlines #19636 bpo-38870: Do not seperate factor prefixes #20133 bpo-38870: don't put unnecessary parantheses on class declarations #20134 bpo-38870: Use subTest in test_unparse for better error reporting #20141 bpo-38870: Only omit slice parantheses when the inner expression is simple  #20152 bpo-38870: fixing unhandled hexescape in docstrings at ast.unparse #20166 bpo-38870: invalid escape sequence #20240 [3.9] bpo-38870: invalid escape sequence (GH-20240) #20244 bpo-38870: Extend subject of ast.unparse warnings #21053 [3.9] bpo-38870: Extend subject of ast.unparse warnings (GH-21053) #21191

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/pablogsal'
closed_at = <Date 2020-10-23.18:24:39.319>
created_at = <Date 2019-11-20.22:34:29.558>
labels = ['type-feature', 'library', '3.9']
title = 'Expose ast.unparse in the ast module'
updated_at = <Date 2020-10-23.18:24:39.319>
user = 'https://github.com/pablogsal'

bugs.python.org fields:

activity = <Date 2020-10-23.18:24:39.319>
actor = 'BTaskaya'
assignee = 'pablogsal'
closed = True
closed_date = <Date 2020-10-23.18:24:39.319>
closer = 'BTaskaya'
components = ['Library (Lib)']
creation = <Date 2019-11-20.22:34:29.558>
creator = 'pablogsal'
dependencies = []
files = []
hgrepos = []
issue_num = 38870
keywords = ['patch']
message_count = 33.0
messages = ['357107', '357111', '357138', '357176', '357258', '357417', '357438', '358478', '358504', '358824', '358826', '359001', '359207', '359500', '359502', '359508', '363087', '363196', '363768', '364257', '367985', '369069', '369070', '369076', '369078', '369088', '369286', '369289', '369398', '369402', '372488', '372490', '379462']
nosy_count = 7.0
nosy_names = ['levkivskyi', 'pablogsal', 'miss-islington', 'brandtbucher', 'BTaskaya', 'Batuhan Taskaya', '1v3m']
pr_nums = ['17302', '17376', '17377', '17612', '17613', '17687', '17738', '17739', '17760', '17797', '17798', '17892', '18768', '19016', '19636', '20133', '20134', '20141', '20152', '20166', '20240', '20244', '21053', '21191']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue38870'
versions = ['Python 3.9']

[pablogsal]

As discussed in https://mail.python.org/archives/list/python-dev@python.org/thread/JAQDBMC23HW2PQ27HQNJ7G244T423IDD/ I propose to expose the unparse.py tool as part of the standard library in the ast module.

The exposed function will have the interface:

ast.unparse(ast_obj, *, option1, option2,...)

and will return a string with the unparsed version of ast_obj.

The unparse tool will need some cosmetic improvements that will be tracked separately in this issue.

[pablogsal]

After PR17302 is merged we need to fix the following cosmetic issues indicated by Victor:

(*) unparse adds many useless parentheses. The algorithm seems naive.
For example, it adds "()" to "class _AddedDllDirectory():". It also
adds parenthesis around yield, like "(yield (top, dirs, nondirs))",
whereas the AST node was at "top level": it isn't a sub-expression.
Maybe this algortihm should be made smarter.

(*) newlines in docstring are rendered as two characters: "\" + "n"
(escaped newline: \n), rather than a newline character. I would expect
a newline, it's more common that \n... But it may "break" inline
doctests rendering... Maybe it should be an option (render newlines as
newline character or escape them?), or maybe even let the user choose
how to render them using a callback (that's related to the "pluggable
formatter" question).

(*) Indentation is hardcoded to 4 spaces. What if I want 2 spaces or
something different? Should it become an option?

(*) Float infinity is replaces with 1e309. Again, maybe someone wants
to render this differently? It sounds like an arbitrary choice (which
"works" as expected).

[BatuhanTaskaya]

After PR 17302 is accepted, I'll work on refactorings including a precedence algorithm to find when to parentheses.

[isidentical]

@gvanrossum are you OK with adding type comments support? Current version loses type comment information so if typed_ast parses this, they wont be the same in AST representation.

[gvanrossum]

@gvanrossum are you OK with adding type comments support? Current version loses type comment information so if typed_ast parses this, they wont be the same in AST representation.

Good catch, definitely do that!

[pablogsal]

New changeset 27fc3b6 by Pablo Galindo in branch 'master':
bpo-38870: Expose a function to unparse an ast object in the ast module (GH-17302)
27fc3b6

[miss-islington]

New changeset ded8888 by Miss Islington (bot) (Pablo Galindo) in branch 'master':
bpo-38870: Remove dependency on contextlib to avoid performance regression on import (GH-17376)
ded8888

[pablogsal]

New changeset a322f50 by Pablo Galindo (Batuhan Taşkaya) in branch 'master':
bpo-38870: Remove dead code related with argument unparsing (GH-17613)
a322f50

[vstinner]

I created bpo-39069: "Move ast.unparse() function to a different module".

[pablogsal]

New changeset 4b3b122 by Pablo Galindo (Batuhan Taşkaya) in branch 'master':
bpo-38870: Refactor delimiting with context managers in ast.unparse (GH-17612)
4b3b122

[pablogsal]

New changeset d69cbeb by Pablo Galindo in branch 'master':
Revert "bpo-38870: Remove dependency on contextlib to avoid performance regression on import (GH-17376)" (GH-17687)
d69cbeb