Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Handle parentheses when formatting slice expressions #5882

Merged
merged 1 commit into from
Jul 19, 2023

Conversation

konstin
Copy link
Member

@konstin konstin commented Jul 19, 2023

Summary Fix the formatter crash with x[(1) :: ] and related code.

Problem For assigning comments in slices in subscripts, we need to find the positions of the colons to assign comments before and after the colon to the respective lower/upper/step node (or dangling in that section). Formatting x[(1) :: ] was broken because we were looking for a : after the 1 but didn't consider that there could be a ) outside the range of the lower node, which contains just the 1 and no optional parentheses.

Solution Use the simple tokenizer directly and skip all closing parentheses.

Test Plan I added regression tests.

Closes #5733

@konstin
Copy link
Member Author

konstin commented Jul 19, 2023

Current dependencies on/for this PR:

This comment was auto-generated by Graphite.

@github-actions
Copy link
Contributor

github-actions bot commented Jul 19, 2023

PR Check Results

Ecosystem

✅ ecosystem check detected no changes.

Benchmark

Linux

group                                      main                                   pr
-----                                      ----                                   --
formatter/large/dataset.py                 1.01      9.9±0.03ms     4.1 MB/sec    1.00      9.8±0.02ms     4.2 MB/sec
formatter/numpy/ctypeslib.py               1.00   1915.5±1.51µs     8.7 MB/sec    1.00   1908.2±2.48µs     8.7 MB/sec
formatter/numpy/globals.py                 1.00    208.1±0.37µs    14.2 MB/sec    1.00    207.1±0.47µs    14.2 MB/sec
formatter/pydantic/types.py                1.00      4.2±0.02ms     6.0 MB/sec    1.00      4.2±0.01ms     6.0 MB/sec
linter/all-rules/large/dataset.py          1.08     14.6±0.03ms     2.8 MB/sec    1.00     13.5±0.02ms     3.0 MB/sec
linter/all-rules/numpy/ctypeslib.py        1.05      3.6±0.00ms     4.6 MB/sec    1.00      3.4±0.00ms     4.9 MB/sec
linter/all-rules/numpy/globals.py          1.03    383.8±1.58µs     7.7 MB/sec    1.00    372.2±1.15µs     7.9 MB/sec
linter/all-rules/pydantic/types.py         1.06      6.5±0.01ms     3.9 MB/sec    1.00      6.1±0.01ms     4.2 MB/sec
linter/default-rules/large/dataset.py      1.15      8.2±0.01ms     5.0 MB/sec    1.00      7.1±0.01ms     5.7 MB/sec
linter/default-rules/numpy/ctypeslib.py    1.12   1610.6±3.96µs    10.3 MB/sec    1.00   1439.3±4.25µs    11.6 MB/sec
linter/default-rules/numpy/globals.py      1.08    162.6±0.26µs    18.1 MB/sec    1.00    150.1±0.88µs    19.7 MB/sec
linter/default-rules/pydantic/types.py     1.13      3.5±0.01ms     7.2 MB/sec    1.00      3.1±0.01ms     8.1 MB/sec

Windows

group                                      main                                   pr
-----                                      ----                                   --
formatter/large/dataset.py                 1.01     11.0±0.15ms     3.7 MB/sec    1.00     10.9±0.16ms     3.7 MB/sec
formatter/numpy/ctypeslib.py               1.01      2.2±0.06ms     7.7 MB/sec    1.00      2.1±0.03ms     7.8 MB/sec
formatter/numpy/globals.py                 1.00    244.8±7.06µs    12.1 MB/sec    1.00    245.1±6.58µs    12.0 MB/sec
formatter/pydantic/types.py                1.00      4.7±0.06ms     5.4 MB/sec    1.00      4.7±0.07ms     5.4 MB/sec
linter/all-rules/large/dataset.py          1.01     15.4±0.14ms     2.6 MB/sec    1.00     15.3±0.16ms     2.7 MB/sec
linter/all-rules/numpy/ctypeslib.py        1.00      4.0±0.04ms     4.2 MB/sec    1.00      4.0±0.06ms     4.1 MB/sec
linter/all-rules/numpy/globals.py          1.00    485.3±8.10µs     6.1 MB/sec    1.00    485.2±9.86µs     6.1 MB/sec
linter/all-rules/pydantic/types.py         1.00      6.9±0.08ms     3.7 MB/sec    1.00      6.9±0.08ms     3.7 MB/sec
linter/default-rules/large/dataset.py      1.01      8.0±0.07ms     5.1 MB/sec    1.00      7.9±0.06ms     5.1 MB/sec
linter/default-rules/numpy/ctypeslib.py    1.01  1659.1±16.22µs    10.0 MB/sec    1.00  1649.8±20.84µs    10.1 MB/sec
linter/default-rules/numpy/globals.py      1.00    189.5±3.81µs    15.6 MB/sec    1.00    190.2±4.49µs    15.5 MB/sec
linter/default-rules/pydantic/types.py     1.00      3.6±0.04ms     7.2 MB/sec    1.00      3.5±0.04ms     7.2 MB/sec

Comment on lines +180 to +182
let mut tokens = SimpleTokenizer::new(contents, TextRange::new(after_upper, range.end()))
.skip_trivia()
.skip_while(|token| token.kind == TokenKind::RParen);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if we should create a helper for this. It's not the first time we incorrectly assumed that there are never right parens. But I don't have a good idea of how, and this is cleary out of this PR's scope.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

An after node range search util would be nice. I did some small changes in #5885

@konstin konstin force-pushed the Fix_slice_formatting_with_parentheses branch from 98e8d42 to e062233 Compare July 19, 2023 14:57
Base automatically changed from add_message_to_syntax_error to main July 19, 2023 15:15
**Summary** Fix the formatter crash with `x[(1) :: ]` and related code.

**Problem** For assigning comments in slices in subscripts, we need to find the positions of the colons to assign comments before and after the colon to the respective lower/upper/step node (or dangling in that section). Formatting `x[(1) :: ]` was broken because we were looking for a `:` after the `1` but didn't consider that there could be a `)` outside the range of the lower node, which contains just the `1` and no optional parentheses.

**Solution** Use the simple tokenizer directly and skip all closing parentheses.

**Test Plan** I added regression tests.

Closes #5733
@konstin konstin force-pushed the Fix_slice_formatting_with_parentheses branch from e062233 to 46b65d0 Compare July 19, 2023 15:16
@konstin konstin enabled auto-merge (squash) July 19, 2023 15:16
@konstin konstin merged commit a51606a into main Jul 19, 2023
15 checks passed
@konstin konstin deleted the Fix_slice_formatting_with_parentheses branch July 19, 2023 15:25
konstin added a commit that referenced this pull request Jul 20, 2023
## Summary

A bit more consistency inspired by
#5882 (comment)

## Test Plan

Existing tests (refactoring)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Subscript formatting fails when element is parenthesized
2 participants