gh-155292: Skip updating unicodedata with mismatched interpreter - #157066
Open
encukou wants to merge 4 commits into
Open
gh-155292: Skip updating unicodedata with mismatched interpreter#157066encukou wants to merge 4 commits into
encukou wants to merge 4 commits into
Conversation
mkstringprep uses things like str.lower(), so it generates the wrong result if run in an interpreter with a different Unicode data version than the target. This means that updating the Unicode version is a two-step process: run makeunicodedata.py, then compile, then run mkstringprep.py. The two steps can (and should) be combined when re-running regen-unicodedata to verify that the data is up to date. The pythonGH-155292 fix only considered that case. Change makeunicodedata.py to only run mkstringprep.py when the current interpreter is up to it. Otherwise, show a reminder. As an extra complication, download the input (RFC 3454) in the "first step", since an out-of-date stringprep.py's freshness assertion may prevent downloads.
encukou
commented
Sep 7, 2026
| data_file = open(FILENAME, encoding='utf-8') | ||
|
|
||
| with data_file: | ||
| with open("Tools/unicode/data/rfc3454.txt", encoding='utf-8') as data_file: |
Member
Author
There was a problem hiding this comment.
I figured we don't need the urllib call here: you shouldn't hit this in the usual case, and if you do, the FileNotFoundError makes it clear that you need rfc3454.txt and where to put it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
mkstringprep uses things like str.lower(), so it generates the wrong
result if run in an interpreter with a different Unicode data version
than the target.
This means that updating the Unicode version is a two-step process:
run makeunicodedata.py, then compile, then run mkstringprep.py.
The two steps can (and should) be combined when re-running
regen-unicodedata to verify that the data is up to date.
The #155292 fix only considered that case.
Change makeunicodedata.py to only run mkstringprep.py when the
current interpreter is up to it. Otherwise, show a reminder.
For an extra complication, download the input (rcf3454.txt) in
the "first step", since an out-of-date stringprep.py's freshness
assertion may break URL encoding in urllib.