Python 3.14 REPL sometimes breaks when editing code blocks

[treyhunner]

Bug report

Bug description:

This has happened to me three times now over the past few weeks and I have not yet been able to reliable reproduce the issue.

The most recent time, I had entered this function into the REPL (typing it all out manually):

>>> def upper_based_on_length(fruit):
...     if len(fruit) < 6:
...         return fruit
...     else:
...         return fruit.upper()
...

I hit the up arrow on my keyboard and I tried to delete the if line with Ctrl+K at the start of the if.

That then immediately showed this error:

>>> def upper_based_on_length(fruit):
...     Traceback (most recent call last):
  File "/home/trey/.pyenv/versions/3.14.0b1/lib/python3.14/_pyrepl/readline.py", line 393, in multiline_input
    return reader.readline()
           ~~~~~~~~~~~~~~~^^
  File "/home/trey/.pyenv/versions/3.14.0b1/lib/python3.14/_pyrepl/reader.py", line 748, in readline
    self.handle1()
    ~~~~~~~~~~~~^^
  File "/home/trey/.pyenv/versions/3.14.0b1/lib/python3.14/_pyrepl/reader.py", line 731, in handle1
    self.do_cmd(cmd)
    ~~~~~~~~~~~^^^^^
  File "/home/trey/.pyenv/versions/3.14.0b1/lib/python3.14/_pyrepl/reader.py", line 661, in do_cmd
    self.refresh()
    ~~~~~~~~~~~~^^
  File "/home/trey/.pyenv/versions/3.14.0b1/lib/python3.14/_pyrepl/reader.py", line 638, in refresh
    self.screen = self.calc_screen()
                  ~~~~~~~~~~~~~~~~^^
  File "/home/trey/.pyenv/versions/3.14.0b1/lib/python3.14/_pyrepl/completing_reader.py", line 261, in calc_screen
    screen = super().calc_screen()
  File "/home/trey/.pyenv/versions/3.14.0b1/lib/python3.14/_pyrepl/reader.py", line 315, in calc_screen
    colors = list(gen_colors(self.get_unicode()))
  File "/home/trey/.pyenv/versions/3.14.0b1/lib/python3.14/_pyrepl/utils.py", line 102, in gen_colors
    for color in gen_colors_from_token_stream(gen, line_lengths):
                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^
  File "/home/trey/.pyenv/versions/3.14.0b1/lib/python3.14/_pyrepl/utils.py", line 160, in gen_colors_from_token_stream
    for prev_token, token, next_token in token_window:
                                         ^^^^^^^^^^^^
  File "/home/trey/.pyenv/versions/3.14.0b1/lib/python3.14/_pyrepl/utils.py", line 357, in prev_next_window
    for x in iterator:
             ^^^^^^^^
  File "/home/trey/.pyenv/versions/3.14.0b1/lib/python3.14/tokenize.py", line 586, in _generate_tokens_from_c_tokenizer
    raise e from None
  File "/home/trey/.pyenv/versions/3.14.0b1/lib/python3.14/tokenize.py", line 582, in _generate_tokens_from_c_tokenizer
    for info in it:
                ^^
  File "<string>", line 4
    else:
         ^
IndentationError: unindent does not match any outer indentation level

Note that I did not try to run this block of code. I was simply editing it. The syntax simply became temporarily invalid while editing.

I have tried to reproduce this in the same REPL environment using a different function name and I can't seem to do so. If I edit the exact code block in history that produced the error, I see the same exception again. However, if I try to rewrite the same function with a different name and then attempt to edit it, I don't see the error again.

CPython versions tested on:

3.14

Operating systems tested on:

Linux

[terryjreedy]

I tried the same in IDLE and when I hit ^K with the cursor just before if, the line disappeared. When I moved the cursor to the end and hit , I got correctly got SyntaxError with the unindent message as above. (For better or worse, IDLE seems to display all syntax errors as SyntaxError.) So the error does not seem to be in the code or codeop modules, which I believe _pyrepl also uses.

[ronaldoussoren]

This could be similar to #133541. Could you check if the error still occurs in the current beta (beta 2)?

[ivanbelenky]

@ronaldoussoren the issue is precisely the same as the cited issue.

If you are using uv and pinning to 3.14 -> 0b1 is the micro installed. I just built 0b2 from source and the bug is impossible to reproduce. If you want to double check you can always go to lib/python3.14/_pyrepl/utils.py and spot that gen_color function is only capturing tokenize.TokenError. You should see

def gen_colors(buffer: str):
    ...
    try:
        for color in gen_colors_from_token_stream(gen, line_lengths):
            yield color
            last_emitted = color
    # this SyntaxError except should be misssing
    except SyntaxError:
        return
    except tokenize.TokenError as te:
        yield from recover_unterminated_string(
            te, line_lengths, last_emitted, buffer
        )

Probably @treyhunner this is worhtless knowing it is fixed in beta 2, but in case you want to reproduce, you can always do something like the cited issue, essentially:

>>> def foo():
...      return 0
... a = 3

Place your cursor delete the a and the error should be triggered. This or whatever construct that involves an indentation token will raise.