formatter panic with comment between joined f-strings

[davidszotten]

a3d4f08 moved the f-string comment handling added in f091b46 (which unfortunately didn't add enough test cases)

(
    f'{1}' # comment
    f'{2}'
)

panics with

Unexpected token between nodes: `"' "`', crates/ruff_python_formatter/src/comments/placement.rs:122:17

[cnpryer]

What's the correct way to read

`"' "`'

[cnpryer]

I imagine the "s are just lexing noise?

ruff/crates/ruff_python_formatter/src/comments/placement.rs

Lines 122 to 126 in fbc9b5a

	debug_assert!(
	!matches!(token.kind, SimpleTokenKind::Bogus),
	"Unexpected token between nodes: `{:?}`",
	locator.slice(TextRange::new(preceding.end(), comment.start()),)
	);

Edit: noise probably isn't a good way to describe it. I mean it's lexed as " tokens or something?

[charliermarsh]

I think the SimpleTokenizer intentionally doesn't support lexing strings, so it hits the first quote and then lexes any subsequent characters as bogus. I'm not quite familiar enough with the f-string comment handling to be sure how to solve this, but I can play around with it for a sec to get an idea.

[davidszotten]

i believe the issue is that the preceding node is the FormattedValue _inside the f-string

the printed slice is

 f'{1}' # comment
      ^^

[davidszotten]

the previous solution was re-assigning the comment from the FormattedValue to the surrounding f-string

[charliermarsh]

Yeah, makes sense. I can probably restore it.

[davidszotten]

i found this when trying to test #6365 after rebasing on top of the refactor in the pr description.