Expose ast.unparse in the ast module

[pablogsal]

BPO	38870
Nosy	@ilevkivskyi, @pablogsal, @miss-islington, @brandtbucher, @isidentical, @CyberSaxosTiGER
PRs	bpo-38870: Expose a function to unparse an ast object in the ast module #17302 bpo-38870: Remove dependency on contextlib to avoid performance regression on import #17376 bpo-38870: Implement Simple Preceding to AST Unparser #17377 bpo-38870: refactor delimiting with context managers #17612 bpo-38870: Remove dead code related with argument unparsing #17613 Revert "bpo-38870: Remove dependency on contextlib to avoid performance regression on import (GH-17376)" #17687 bpo-38870: Run always tests that heavily use grammar features in test_unparse #17738 bpo-38870: Fix error when running with -uall in test_unparse #17739 bpo-38870: Docstring support for function/class/module nodes #17760 bpo-38870: Implement round tripping support for typed AST #17797 bpo-38870: Throw ValueError on invalid yield from usage #17798 bpo-38870: Simplify tuple like interleaves, roundtrip extslice properly #17892 bpo-38870: Correctly unparse empty docstrings #18768 bpo-38870: Implement support for ast.FunctionType #19016 bpo-38870: Don't start generated output with newlines #19636 bpo-38870: Do not seperate factor prefixes #20133 bpo-38870: don't put unnecessary parantheses on class declarations #20134 bpo-38870: Use subTest in test_unparse for better error reporting #20141 bpo-38870: Only omit slice parantheses when the inner expression is simple  #20152 bpo-38870: fixing unhandled hexescape in docstrings at ast.unparse #20166 bpo-38870: invalid escape sequence #20240 [3.9] bpo-38870: invalid escape sequence (GH-20240) #20244 bpo-38870: Extend subject of ast.unparse warnings #21053 [3.9] bpo-38870: Extend subject of ast.unparse warnings (GH-21053) #21191

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/pablogsal'
closed_at = <Date 2020-10-23.18:24:39.319>
created_at = <Date 2019-11-20.22:34:29.558>
labels = ['type-feature', 'library', '3.9']
title = 'Expose ast.unparse in the ast module'
updated_at = <Date 2020-10-23.18:24:39.319>
user = 'https://github.com/pablogsal'

bugs.python.org fields:

activity = <Date 2020-10-23.18:24:39.319>
actor = 'BTaskaya'
assignee = 'pablogsal'
closed = True
closed_date = <Date 2020-10-23.18:24:39.319>
closer = 'BTaskaya'
components = ['Library (Lib)']
creation = <Date 2019-11-20.22:34:29.558>
creator = 'pablogsal'
dependencies = []
files = []
hgrepos = []
issue_num = 38870
keywords = ['patch']
message_count = 33.0
messages = ['357107', '357111', '357138', '357176', '357258', '357417', '357438', '358478', '358504', '358824', '358826', '359001', '359207', '359500', '359502', '359508', '363087', '363196', '363768', '364257', '367985', '369069', '369070', '369076', '369078', '369088', '369286', '369289', '369398', '369402', '372488', '372490', '379462']
nosy_count = 7.0
nosy_names = ['levkivskyi', 'pablogsal', 'miss-islington', 'brandtbucher', 'BTaskaya', 'Batuhan Taskaya', '1v3m']
pr_nums = ['17302', '17376', '17377', '17612', '17613', '17687', '17738', '17739', '17760', '17797', '17798', '17892', '18768', '19016', '19636', '20133', '20134', '20141', '20152', '20166', '20240', '20244', '21053', '21191']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue38870'
versions = ['Python 3.9']

[pablogsal]

As discussed in https://mail.python.org/archives/list/python-dev@python.org/thread/JAQDBMC23HW2PQ27HQNJ7G244T423IDD/ I propose to expose the unparse.py tool as part of the standard library in the ast module.

The exposed function will have the interface:

ast.unparse(ast_obj, *, option1, option2,...)

and will return a string with the unparsed version of ast_obj.

The unparse tool will need some cosmetic improvements that will be tracked separately in this issue.

[pablogsal]

After PR17302 is merged we need to fix the following cosmetic issues indicated by Victor:

(*) unparse adds many useless parentheses. The algorithm seems naive.
For example, it adds "()" to "class _AddedDllDirectory():". It also
adds parenthesis around yield, like "(yield (top, dirs, nondirs))",
whereas the AST node was at "top level": it isn't a sub-expression.
Maybe this algortihm should be made smarter.

(*) newlines in docstring are rendered as two characters: "\" + "n"
(escaped newline: \n), rather than a newline character. I would expect
a newline, it's more common that \n... But it may "break" inline
doctests rendering... Maybe it should be an option (render newlines as
newline character or escape them?), or maybe even let the user choose
how to render them using a callback (that's related to the "pluggable
formatter" question).

(*) Indentation is hardcoded to 4 spaces. What if I want 2 spaces or
something different? Should it become an option?

(*) Float infinity is replaces with 1e309. Again, maybe someone wants
to render this differently? It sounds like an arbitrary choice (which
"works" as expected).

[BatuhanTaskaya]

After PR 17302 is accepted, I'll work on refactorings including a precedence algorithm to find when to parentheses.

[isidentical]

@gvanrossum are you OK with adding type comments support? Current version loses type comment information so if typed_ast parses this, they wont be the same in AST representation.

[gvanrossum]

@gvanrossum are you OK with adding type comments support? Current version loses type comment information so if typed_ast parses this, they wont be the same in AST representation.

Good catch, definitely do that!

[pablogsal]

New changeset 27fc3b6 by Pablo Galindo in branch 'master':
bpo-38870: Expose a function to unparse an ast object in the ast module (GH-17302)
27fc3b6

[miss-islington]

New changeset ded8888 by Miss Islington (bot) (Pablo Galindo) in branch 'master':
bpo-38870: Remove dependency on contextlib to avoid performance regression on import (GH-17376)
ded8888

[pablogsal]

New changeset a322f50 by Pablo Galindo (Batuhan Taşkaya) in branch 'master':
bpo-38870: Remove dead code related with argument unparsing (GH-17613)
a322f50

[vstinner]

I created bpo-39069: "Move ast.unparse() function to a different module".

[pablogsal]

New changeset 4b3b122 by Pablo Galindo (Batuhan Taşkaya) in branch 'master':
bpo-38870: Refactor delimiting with context managers in ast.unparse (GH-17612)
4b3b122

[pablogsal]

New changeset d69cbeb by Pablo Galindo in branch 'master':
Revert "bpo-38870: Remove dependency on contextlib to avoid performance regression on import (GH-17376)" (GH-17687)
d69cbeb

[pablogsal]

New changeset 23a226b by Pablo Galindo in branch 'master':
bpo-38870: Run always tests that heavily use grammar features in test_unparse (GH-17738)
23a226b

[pablogsal]

New changeset 7b35bef by Pablo Galindo (Batuhan Taşkaya) in branch 'master':
bpo-38870: Throw ValueError on invalid yield from usage (GH-17798)
7b35bef

[isidentical]

ExtSlice nodes without second value doesn't roundtrip properly

source: x[1:2,]
         Expr(
             value=Subscript(
                 value=Name(id='x', ctx=Load()),
-                slice=ExtSlice(
-                    dims=[
-                        Slice(
-                            lower=Constant(value=1, kind=None),
-                            upper=Constant(value=2, kind=None),
-                            step=None)]),
+                slice=Slice(
+                    lower=Constant(value=1, kind=None),
+                    upper=Constant(value=2, kind=None),
+                    step=None),
                 ctx=Load()))],
     type_ignores=[])

(I have a fix for unifying both tuple, constant tuple and extslice dims traversing)

[isidentical]

We might need to tweak the documentation @pablogsal,

Unparse an ast.AST object and generate a string with code that would produce an equivalent ast.AST object if parsed back with ast.parse().

If I interpret equivalent correctly, this explanation is false under cases like constant nodes has an immutable container value (which they can).

>>> def wrap(expr):
...     return ast.Module(body=[ast.Expr(expr)], type_ignores=[])
... 
>>> constant_tuple = wrap(ast.Constant(value=(1, 2, 3), kind=None))
>>> normalpy_tuple = ast.parse("(1, 2, 3)")
>>> ast.unparse(constant_tuple) == ast.unparse(normalpy_tuple)
True
>>> ast.dump(ast.parse(ast.unparse(constant_tuple))) != ast.dump(constant_tuple)
True
>>> ast.dump(ast.parse(ast.unparse(constant_tuple))) == ast.dump(normalpy_tuple)
True

This isn't a bug in the code because there is no way we can generate a string that can produce a constant tuple. But this makes the docstring false, at least in certain cases.

[pablogsal]

We might need to tweak the documentation @pablogsal,

Maybe we can say that is 'as close as possible' if you pass an arbitrary AST object (if only happens with Constants we could documment exactly that). Let me think about this and I will make PR updating the docs.

[pablogsal]

New changeset 397b96f by Batuhan Taşkaya in branch 'master':
bpo-38870: Implement a precedence algorithm in ast.unparse (GH-17377)
397b96f

[pablogsal]

New changeset 89aa469 by Batuhan Taşkaya in branch 'master':
bpo-38870: Add docstring support to ast.unparse (GH-17760)
89aa469

[pablogsal]

New changeset e7cab7f by Batuhan Taşkaya in branch 'master':
bpo-38870: Simplify sequence interleaves in ast.unparse (GH-17892)
e7cab7f

[pablogsal]

New changeset 5b66ec1 by Batuhan Taşkaya in branch 'master':
bpo-38870: Implement support for ast.FunctionType in ast.unparse (GH-19016)
5b66ec1

[pablogsal]

New changeset 493bf1c by Batuhan Taskaya in branch 'master':
bpo-38870: Don't start generated output with newlines in ast.unparse (GH-19636)
493bf1c

[pablogsal]

New changeset ce4a753 by Batuhan Taskaya in branch 'master':
bpo-38870: Do not separate factor prefixes in ast.unparse (GH-20133)
ce4a753

[pablogsal]

New changeset 25160cd by Batuhan Taskaya in branch 'master':
bpo-38870: Don't put unnecessary parentheses on class declarations in ast.parse (GH-20134)
25160cd

[pablogsal]

New changeset e966af7 by Batuhan Taskaya in branch 'master':
bpo-38870: Correctly handle empty docstrings in ast.unparse (GH-18768)
e966af7

[pablogsal]

New changeset dff92bb by Batuhan Taskaya in branch 'master':
bpo-38870: Implement round tripping support for typed AST in ast.unparse (GH-17797)
dff92bb

[pablogsal]

New changeset 6341fc7 by Pablo Galindo in branch 'master':
bpo-38870: Use subTest in test_unparse for better error reporting (GH-20141)
6341fc7

[pablogsal]

New changeset d71a649 by CyberSaxosTiGER in branch 'master':
bpo-38870: correctly escape unprintable characters on ast.unparse (GH-20166)
d71a649

[pablogsal]

New changeset c102a14 by Batuhan Taskaya in branch 'master':
bpo-38870: Don't omit parenthesis when unparsing a slice in ast.unparse
c102a14

[miss-islington]

New changeset dd74b6f by Batuhan Taskaya in branch 'master':
bpo-38870: invalid escape sequence (GH-20240)
dd74b6f

[miss-islington]

New changeset 059279d by Miss Islington (bot) in branch '3.9':
bpo-38870: invalid escape sequence (GH-20240)
059279d

[pablogsal]

New changeset 8df1016 by Batuhan Taskaya in branch 'master':
bpo-38870: Extend subject of ast.unparse warnings (GH-21053)
8df1016

[pablogsal]

New changeset 6e39999 by Pablo Galindo in branch '3.9':
[3.9] bpo-38870: Extend subject of ast.unparse warnings (GH-21053) (GH-21191)
6e39999

[isidentical]

Well, I'm happy to say that 3.9.0 is finally released with the ast.unparse interface. After tens of PRs, I think it is time for us to move on. For all future bugs, please create a new issue and nosy me. Thanks everyone who has participated this journey!