In IDLE colorizer, replace re with tokenizer

[terryjreedy]

The current IDLE colorizer was designed when the python grammar was restricted to a context-free subtype. Parsing with RE's was sufficient or closely so to pick out substrings to be color tagged. The current grammar's sometime context-dependence make this more difficult or even impossible.

A few versions ago, the new C tokenizer used to compile Python code was exposed as a function in the tokenizer module. It replaced a python-coded tokenizer that did not always match the old C tokenizer and that must have been much slower. I assume that is was not seen as suitable for IDLE colorizer. However, the current exposed C tokenizer is being used to power the new PyREPL colorizer. The initial (draft) patch copies portions of that colorizer.

My main concerns are compatibility and speed. Some initial questions:

1. The syntax categories in the REPL colorizer are not the same as IDLE's (listed on the Highlights settings page). What are they?
2. IDLE colorizes 10000 lines editor contents as well as single interactive lines. Will the PR colorizer do the same, with similar speed?
3. Is REPL colorizer stable? Are there open bug reports? (Should be labelled topic-repl.) The code is private and undocumented, I presume intentionally, and I do not expect that issues only relevant to IDLE would be welcome.
4. In the IDLE Issues project, colorizer issues are in Highlights section. Does this fix any, or appear to make fixes easier?

Linked PRs

• gh-140347: Use pyrepl's colorizer in IDLE #140337

[StanFromIreland]

Starting from the bottom.

In the IDLE Issues project, colorizer issues are in Highlights section. Does this fix any, or appear to make fixes easier?

• IDLE needs syntax highlighting for f-strings #73473 is fixed.
• IDLE: Wrong highlighting when previous line ends with backslash #140334 is also fixed.

The remainder of the issues in the project are not directly related to the actual colorizer logic, and there should be no change in how they would be implemented.

IDLE colorizes 10000 lines editor contents as well as single interactive lines. Will the PR colorizer do the same, with similar speed?

The PR should be faster than the regexes, although I have not yet done proper benchmarking.

The syntax categories in the REPL colorizer are not the same as IDLE's (listed on the Highlights settings page). What are they?

They can be modified to match, and in the PR it does.

The last one requires more research, for which I do not have time right now.

[StanFromIreland]

My main issue with the implementation is that the old colorizer would continue on errornous syntax, whereas tokenizing may fail. The repl colorizer has a fallback, but it does not always succeed. Trying to complicate the logic more is not worth it IMO, I have meddled with it a little and introduced a second line-by-line attempt, though that also causes problems (I will try fix these, the main issue with line-by-line is context, e.g. are we in a triple quoted string, is lost).
I think we would need to accept that on errornous syntax, there sometimes may be no highlighting at all.