Prepare for future normalization data for Unicode 16 #4538
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ICU4X's data representation doesn't have a place for marking the first character of an expansion as a starter that combines backwards. To accommodate for the Unicode 16 characters discussed in section 5 of https://www.unicode.org/L2/L2024/24009-utc178-properties-recs.pdf I expect the least disruptive change to be that characters that can occur as the first character of a composition whose second character may occur as the first character of an expansion decomposition be represented the way non-BMP singleton decompositions are currently represented. This should exclude them from NFC passthrough eligibility without having to find a way to mark the second character. Without this patch, these characters would be incorrectly be claimed as having a singleton decomposition (to self) in `CanonicalDecomposition::decompose()`. There are no tests, because we don't actually have data like this, yet. The purpose is to enable a copy of ICU4X binary that has this code to be able to consume dynamically-loaded data from the future. This change is low-risk in case we don't end up using the data trick anticipated by this patch.
|
The analogous case could be supported for BMP data as well. However, the upcoming characters that need this are non-BMP characters and a BMP character could be represented in the part of the data intended for non-BMP characters. |
|
@hsivonen OK to merge? |
Manishearth
approved these changes
Mar 7, 2024
|
Thanks. It looks like https://unicode-org.atlassian.net/browse/ICU-22586 is still open, so running the data pipeline with Unicode 16 data remains blocked. |
Yes. I plan to get to that before the end of this month. I have a couple of things lined up before that. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
ICU4X's data representation doesn't have a place for marking the first character of an expansion as a starter that combines backwards.
To accommodate the Unicode 16 characters discussed in section 5 of https://www.unicode.org/L2/L2024/24009-utc178-properties-recs.pdf I expect the least disruptive change to be that characters that can occur as the first character of a composition whose second character may occur as the first character of an expansion decomposition be represented the way non-BMP singleton decompositions are currently represented. This should exclude them from NFC passthrough eligibility without having to find a way to mark the second character.
Without this patch, these characters would be incorrectly be claimed as having a singleton decomposition (to self) in
CanonicalDecomposition::decompose().There are no tests, because we don't actually have data like this, yet. The purpose is to enable a copy of ICU4X binary that has this code to be able to consume dynamically-loaded data from the future.
This change is low-risk in case we don't end up using the data trick anticipated by this patch.