Split/Remove Backward tokenization

[MichaReiser]

The SimpleTokenizer supports backward lexing. The implementation will break when supporting Python 3.12's new F-String parsing because comments can now appear in parts that appear to be strings.

f"test{more  # quote
}"

My preferred option would be to remove backward lexing altogether. But I'm unsure how to support is_parenthesized_expression without it in the formatter. The problem is that we need to look back from the start of the expression. One option I could think of is to integrate the parenthesize detection into the CommentsVisitor where we already track the parent nodes (necessary to avoid mistaking a as a parenthesized expression in call(a)), and we could store the last start/position and start lexing from there. This would require skipping over some tokens, which I'm not sure we can handle.

The other alternative is to split the SimpleTokenizer into one implementation that only supports forward lexing and a BackwardTokenizer that supports backward lexing, but takes the CommentRanges as a second argument.

CC: @dhruvmanila

[dhruvmanila]

/cc @konstin who's working on this, thanks for taking this up!

[charliermarsh]

I believe this landed.