Fix lexing single-quoted f-string with multi-line format spec

[dhruvmanila]

Summary

Reported at python/cpython#110259

Test Plan

Add test cases for the fix and update the snapshots

[dhruvmanila]

Current dependencies on/for this PR:

• main
  • PR Fix lexing single-quoted f-string with multi-line format spec #7787 👈

This comment was auto-generated by Graphite.

[codspeed-hq]

CodSpeed Performance Report

Merging #7787 will improve performances by 2.5%

_{Comparing dhruv/multiline-fstring-with-format-spec (9e6ae10) with main (17fba99)}

Summary

⚡ 1 improvements
✅ 24 untouched benchmarks

Benchmarks breakdown

	Benchmark	main	dhruv/multiline-fstring-with-format-spec	Change
⚡	linter/default-rules[pydantic/types.py]	38.9 ms	38 ms	+2.5%

[github-actions]

PR Check Results

Ecosystem

✅ ecosystem check detected no changes.

[dhruvmanila]

The CPython PR has been merged: python/cpython#110271

[pablogsal]

Friendly reminder to review the discussion in python/cpython#110259 so you end implementing the same as CPython 😉

[dhruvmanila]

Friendly reminder to review the discussion in python/cpython#110259 so you end implementing the same as CPython 😉

I did, thanks for the heads up! :)

[dhruvmanila]

Actually, there's more to it for single-quoted f-string that I missed (sorry!). The newline should basically end the lexing of format spec which we currently don't.

[pablogsal]

Actually, there's more to it for single-quoted f-string that I missed (sorry!). The newline should basically end the lexing of format spec which we currently don't.

That's why I suggested to double check :)

[dhruvmanila]

That's why I suggested to double check :)

Hehe, I must've missed that part 😅

---

I'm going to try a couple solutions and look at the benchmarks, please bear with me :)

[dhruvmanila]

It's important that b is a Name token as we ended the format spec lexing at the newline. This will help the parser to raise a syntax error (unexpected token 'b') similar to the CPython implementation.

f'__{
	x:a
		b
}__'

[dhruvmanila]

Update after the first review is that the following case will now raise a syntax error as it should:

f'__{
	x:a
		b
}__'

The reason it didn't earlier is because we didn't end the format spec lexing after we reached the newline (after a). Now, we do which means the lexer will emit a Name token for b for which the parser will raise an unexpected token error.