gh-140347: WIP spike: color IDLE with the _pyrepl tokenizer#152756
Draft
serhiy-storchaka wants to merge 1 commit into
Draft
gh-140347: WIP spike: color IDLE with the _pyrepl tokenizer#152756serhiy-storchaka wants to merge 1 commit into
serhiy-storchaka wants to merge 1 commit into
Conversation
Proof of concept, not for merge. Use _pyrepl.utils.gen_colors in ColorDelegator._add_tags_in_section instead of the regexp, keeping the regexp only for SYNC boundaries. Correct on valid code, but 3 colorizer tests fail by design: gen_colors stops at the first TokenError, blanking the rest of a section. See the issue.
| import re | ||
| import time | ||
|
|
||
| from _pyrepl.utils import gen_colors |
Member
There was a problem hiding this comment.
This import has to be optional, as Steve did for all the others some time ago. That's why I proposed moving it to _colorize.
Member
Author
There was a problem hiding this comment.
Will update. This is just a draft for now.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Draft / proof-of-concept for gh-140347 — not for merge as-is.
This is the Stage-1 spike from my analysis on the issue: it gives the discussion working code to react to.
What it does
Replace the regexp in
ColorDelegator._add_tags_in_sectionwith_pyrepl.utils.gen_colors(the tokenizer-based colorizer that powers the PyREPL). IDLE's incremental engine is untouched —notify_range→TODO→recolorize_mainwalkingSYNCboundaries still drives everything; only the per-section coloring changes. The regexp is kept solely to markSYNCboundaries.gen_colors' categories map cleanly onto IDLE's tags (pyrepl is a superset — it colors builtins too); onlynumber/opare extra and left uncolored. On valid code this is strictly more correct:match/case/typesoft keywords and f-strings the regexp can't handle.Why it's a draft
Three
test_colorizertests fail by design:gen_colorsstops at the firstTokenError, so one invalid token (e.g. an illegal string prefix) blanks the rest of a section, where the regexp used to continue. IDLE amplifies this becauserecolorize_mainaccumulates largeSYNC-bounded sections.The real fix is
TokenErrorrecovery in the sharedgen_colors(restart tokenizing after a bad line), so PyREPL and IDLE both benefit rather than IDLE reintroducing regexp logic. That, plus the "un-tokenizable line" UX call, is the open question on the issue.🤖 Generated with Claude Code