Bug with the colorizer dependent on tokenizer

[Locked-chess-official]

Bug report

Bug description:

wrong_str = "
123456
123456
"

It is a SyntaxError. However, in many web page, the 123456 are colored as string.
This question appeared also in REPL. The IDLE had no this question originally, but after using tokenizer to color the codes, it appears.

It may mislead the learners.

Now that the github page has no question on it, but discuss has.

CPython versions tested on:

CPython main branch, 3.14

Operating systems tested on:

Windows

[StanFromIreland]

Note that this is just an idea for IDLE, not something that has been implemented.

[zahlman]

This is about the REPL; IDLE works according to OP's expectations.

In either case, the code has to be pasted in to demonstrate the problem — otherwise, the interpreter will report a syntax error as soon as the Enter key is pressed after the opening quotation mark.

However, in IDLE the invalid pasted code will show up with invalid highlighting: the intended contents of the string literal have the default colour rather than string-literal colour. (The opening quote will, however, only be highlighted after pressing Enter.)

At the REPL, meanwhile, the pasted code is coloured as if it were a valid string literal.

I'm not sure that this is particularly important, since proper feedback is almost immediate and even delaying that feedback a little bit requires copying and pasting invalid code from elsewhere. However, it's a strange inconsistency.

As for syntax highlighting on web pages, that is obviously outside of CPython's control. Every web page author can make the code appear with whatever styling is desired, even if it's misleading, and there's nothing that CPython could do about that. (As far as I'm aware, the CPython documentation doesn't show invalid string literals like this, except specifically to explain that they are invalid.)

[terryjreedy]

While there may be a preference for how syntax erroneous code is colorized, there is no correct way. I believe @ambv has said something similar. There is no violation of even an implied specification. Hence, I do not see a bug here and am removing the bug label.

That said, I cannot reproduce the claimed REPL behavior. After copying the code above and running with the same binary, each line is run separately, as with the old REPL.

PS C:\Users\Terry> py -3.14
Python 3.14.0 (tags/v3.14.0:ebf955d, Oct  7 2025, 10:15:03) [MSC v.1944 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> wrong_str = "
  File "<python-input-0>", line 1
    wrong_str = "
                ^
SyntaxError: unterminated string literal (detected at line 1)
>>> 123456
123456
>>> 123456
123456
>>>  # Trailing " becomes a space. 

Ditto in Command Prompt with 3.15 (main). Is there a way to change REPL paste behavior to be more like IDLE's?

[Locked-chess-official]

After copying the code above and running with the same binary, each line is run separately, as with the old REPL.

May be use "Ctrl + shift + V"?

Another way is here:

• Input "wrong_str=''' "
• Enter
• Finally "↑" and delete the '' at the first line.

However, now the way to draw the code makes this code look "valid". Maybe it is not a good idea.

[Locked-chess-official]

In fact, there is another problem in it: the syntax label is unstable in this absolute invalid syntax.

For example, I wrote the code, but I want to print a string before to debug it. After I input a " or ' the code after were change its color. This effect is bad.

[StanFromIreland]

That said, I cannot reproduce the claimed REPL behavior. After copying the code above and running with the same binary, each line is run separately, as with the old REPL.

On Linux, it is not run till one presses Enter, so I can reproduce the reported colorizing.

[picnixz]

It's not just for strings that we have this disparity. For instance, on Linux:

wrong = from x import (
  1 + 1
)

In the REPL, I get (before hitting ENTER):

and once I hit ENTER, I will have a SyntaxError. On the other hand, in IDLE (before hitting ENTER), I get:

When someone C/C in the REPL, it gets colorized independently of whether it's legit code or not, while in IDLE, it gets colorized but not everything. I don't know if it's possible to have a common implementation for colorizing inputs by REPL/IDLE before hitting ENTER (I would say yes since they should only depend on the input lines that could be tokenizied in one go and then colorized).

In some situations, it makes sense to keep highlights (for instance, in my example, if I were to change 1 + 1 to a legitimate import, but keep wrong =, with colorization, I would see that wrong = should be removed but without it, it would be less explicit).

[StanFromIreland]

I don't know if it's possible to have a common implementation for colorizing inputs by REPL/IDLE

We are working on that:-) See the PR linked in the issue description.

[zahlman]

I don't know if it's possible to have a common implementation for colorizing inputs by REPL/IDLE

We are working on that:-) See the PR linked in the issue description.

I guess you're referring to #140337, now linked from #140347 (not here — until now ;) )?

[StanFromIreland]

Apologies I was on mobile and I could not provide a link, but this hyperlink the original post leads me to it?

but after using tokenizer to color the codes, it appears.