Make pasta indentation easier. #97
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
pasta guarantees lossless round-trip but doesn't provide an easy way to change indentation levels. This change adds a special @@indent@@ token which is embedded into the contents of pasta's annotations dict whenever whitespace is detected to be for the purpose of indentation. It is stored instead of the indentation whitespace is represents. If the 'indent' dict entry is changed, the generated code's indentation level will change as a result.
Recursive calls in to_tree_str had the wrong argument order.
Block whitespace uses the token generator directly. Therefore, factor out the @@nl@@ to @@indent@@ logic from ws() and also apply it wherever block_suffix_* is set. Clean up the logic for @block_statement: Make a is_annotator property on the Annotator classes. Inline block_suffix, which is confusing, and use the same logic we use in other places where we read block_whitespace from the tokenizer. Fix tests and add a one for block whitespace indentation handling.
Now with indentation handled explicitly, the previous practice of indenting once per with, even in the continued case, is shown to be buggy. This might have shown up in a test that tests automatic indentation after a continued with, but we have no such test. This commit fixes the visitation such that a continued with statement is visited without an indented wrapper.
try...except...finally is handled in 2.7 with a TryFinally containing a TryExcept as its only body node. Because the inner TryExcept does not absorb any tokens, it "steals" the whitespace from its first body node. That in turn leads to the indent_diff and indent being computed wrongly, and messes up everything else. To fix, don't use @block_statement on TryExcept, and call prefix only if the statement is not continued. Continued statements don't represent any tokens, and therefore should't have any prefix.
is_continued (for 2.7 try/with statements) was added as a node attribute. This moves the information to pasta's formatting dict, which is cleaner (and allows to_tree_str to print the information).
soupytwist
reviewed
Dec 8, 2021
| # Replace @@NL@@<indent> with @@indent@@. Note this string munging is safe | ||
| # because inside whitespace, @@indent@@ is never legal after a newline | ||
| # (only after a comment, which needs a '#' after a newline). | ||
| # This might not work in the presence of '\' line continuations. |
There was a problem hiding this comment.
Is this still true / okay? Is there a TODO here?
There was a problem hiding this comment.
Ah. I haven't tested, and I don't think that there are any tests that consider \ continuations. So I expect this to not work in general right now.
The tokenizer will not emit anything for a '', not even in the .src attribute of the tokens:
"this is a" + \
"continued line"
turns into
1,0-1,11: STRING '"this is a"'
1,12-1,13: OP '+'
2,0-2,16: STRING '"continued line"'
2,16-2,17: NEWLINE '\n'
3,0-3,1: NL '\n'
4,0-4,0: ENDMARKER ''
So I think handling this properly might be infeasible?
There was a problem hiding this comment.
Interestingly, I think this means that ''.join([t.src for t in tokenize.generate_tokens(src)]) != src. That pretty much guarantees that none of the usual guarantees pasta makes will hold in the presence of ''.
There was a problem hiding this comment.
This is fixable if we want to add back the '\\n' into the whitespace in token_generator. I think that might work, but I'm not sure it's worth doing.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
pasta guarantees lossless round-trip but doesn't provide an easy way to change indentation levels. This change adds a special @@indent@@ token which is embedded into the contents of pasta's annotations dict whenever whitespace is detected to be for the purpose of indentation. It is stored instead of the indentation whitespace is represents. If the 'indent' dict entry is changed, the generated code's indentation level will change as a result.