generate_tokens tokenizes $ differently with Python 3.12 than earlier

[pekkaklarck]

I tested Python 3.12 beta 1 with Robot Framework and noticed that tokenize.generate_tokens() handles expressions containing $ differently than earlier. Earlier $ yielded ERRORTOKEN but nowadays we get OP:

Python 3.11.3 (main, Apr  5 2023, 14:15:06) [GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from tokenize import generate_tokens
>>> from io import StringIO
>>> next(generate_tokens(StringIO('$x').readline))
TokenInfo(type=60 (ERRORTOKEN), string='$', start=(1, 0), end=(1, 1), line='$x')

Python 3.12.0b1 (main, May 22 2023, 23:31:26) [GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from tokenize import generate_tokens
>>> from io import StringIO
>>> next(generate_tokens(StringIO('$x').readline))
TokenInfo(type=55 (OP), string='$', start=(1, 0), end=(1, 1), line='$x\n')

We support Python evaluation with special variables like $var > 1 in Robot Framework data and this change breaks our tokenizing code. I didn't notice anything related in the release notes and decided to report this. If the change is intentional, we can easily update our code to handle also these semantics.

Notice also that there's a small change with TokenInfo.line above. With Python 3.12 there's an additional \n even though the original string didn't contain any newlines.

[sunmy2019]

I didn't notice anything related in the release notes

It just makes it to the deadline. I think the docs can be updated later.
#104323 (comment)

[pekkaklarck]

If I understand the referenced comment correctly, generate_tokens nowadays uses a totally different tokenizer underneath and there can be subtle differences like this one. If that's the case, this likely won't be considered a bug and the old behavior won't be restored. That's fine, I can update our code, but mentioning this under backwards incompatible changes in the release notes would be a good idea.

[pablogsal]

Thanks for raising this with us! I can confirm that this is a expected side effect of the change. Docs are being worked on here:

#104824

I am closing this issue as we will cover the docs part on the linked PR.

[pablogsal]

Opened #104825 for the new line character