gh-140334: Fix IDLE syntax highlighting after line continuations

[TheLizzard]

Description

This PR fixes incorrect syntax highlighting in IDLE when a line ends with a backslash (\) used for line continuation.

Previously

Identifiers following a backslash were sometimes misinterpreted as keywords or built-ins instead of variables or attributes.
For example in:

x = match if match else \
    match

The last match was coloured like a keyword instead of a variable.

Summary of changes

• Updated IDLE's syntax highlighting to properly treat backslash line continuations
• Added tests for cases involving identifiers following a line continuation backslash

Fixes gh-140334

[python-cla-bot]

All commit authors signed the Contributor License Agreement.

[StanFromIreland]

I can confirm the fix works on Linux:

But during typing that I noticed that the keyword pattern also suffers from this issue:

I think this regex approach is getting very messy, the builtin get's quite a bit more complex. I assume that brings significant performance penalties. At a minium I would suggest refactoring some things, rather than having to explain the lookahead each time move it to some constant e.g. LAST_LINE_NO_CONTINUATION. At best I think we could possibly try an reuse the new repl's logic, which uses tokens at not such excessive regexs.

[StanFromIreland]

This pattern also suffers from the issue.

[TheLizzard]

Since the code str.for will always raise a SyntaxError, colouring in the for like a keyword will help beginners recognise that keywords are reserved and cannot be used as variable names/attributes. I believe that the keyword pattern is working correctly. I updated the match/case patterns since they are soft keywords and are valid variable/attribute names.

[StanFromIreland]

Please do not use the Update Branch button unless necessary (e.g. fixing conflicts, jogging the CI, or very old PRs) as it uses valuable resources. For more information see the devguide.

[TheLizzard]

I did some profiling and here are my results:

• Parsing and iterating over the text tags, IDLE can parse ~46k lines a second.
• Parsing and adding the tags to the tkinter.Text widget, IDLE can parse ~10k lines a second.
  Both of these measurements were made parsing Lib/tkinter/__init__.py (since it's nearly 5k lines long).

I think that re-writing it with _pyrepl will improve the highlighting (eg. nested f-strings) but won't improve the speed by a lot (since tkinter is the bottleneck). I haven't used _pyrepl but since it was made for a REPL, I don't know if it will create more bugs in IDLE than solve. I can write more test cases since IDLE is sorely lacking in that department.

[TheLizzard]

diff --git a/Lib/idlelib/colorizer.py b/Lib/idlelib/colorizer.py
index b4df353012..5eb06581ec 100644
--- a/Lib/idlelib/colorizer.py
+++ b/Lib/idlelib/colorizer.py
@@ -273,6 +273,20 @@ def recolorize(self):
 
     def recolorize_main(self):
         "Evaluate text and apply colorizing tags."
+
+        # Shortcut for when we are recoloring everything
+        # Usually this happens when we open a file
+        todo_tag_range = self.tag_nextrange("TODO", "1.0")
+        if not todo_tag_range:
+            return None
+        start, end = todo_tag_range
+        if (start == "1.0") and self.compare(end, "==", "end"):
+            # Remove all tags
+            self.removecolors()
+            # Recolor everything
+            self._add_tags_in_section(self.get("1.0", "end"), "1.0")
+            return None
+
         next = "1.0"
         while todo_tag_range := self.tag_nextrange("TODO", next):
             self.tag_remove("SYNC", todo_tag_range[0], todo_tag_range[1])
@@ -340,14 +354,22 @@ def _add_tags_in_section(self, chars, head):
 
             `head` is the index in the text widget where the text is found.
         """
-        for m in self.prog.finditer(chars):
-            for name, matched_text in matched_named_groups(m):
-                a, b = m.span(name)
-                self._add_tag(a, b, head, name)
+        subtract_chars = 0
+        for match in self.prog.finditer(chars):
+            for name, matched_text in matched_named_groups(match):
+                start, end = match.span(name)
+                self._add_tag(start-subtract_chars, end-subtract_chars, head,
+                              name)
                 if matched_text in ("def", "class"):
-                    if m1 := self.idprog.match(chars, b):
-                        a, b = m1.span(1)
-                        self._add_tag(a, b, head, "DEFINITION")
+                    if new_match := self.idprog.match(chars, end):
+                        start, end = new_match.span(1)
+                        self._add_tag(start-subtract_chars, end-subtract_chars,
+                                      head, "DEFINITION")
+                # Move the head to where `end` points to
+                # Also keep track of how far we've moved head so that we
+                # can account for that next time we call `self._add_tag`
+                head = self.index(f"{head} +{end-subtract_chars:d}c")
+                subtract_chars = end
 
     def removecolors(self):
         "Remove all colorizing tags."

Usually IDLE takes 0.6 seconds to recolorize Lib/tkinter/__init__.py but with the patch above (applies to the main branch), it takes 0.3 seconds. The patch adds 11 lines of code and doubles the speed when colorizing large files.

I would make this into its own separate PR but all PRs need GitHub issues and I don't know how to phrase the issue. I pushed the change to my forked cpython if anyone is interested.