Python 3: When processing xml coming from virtual buffers, make sure that surrogate characters are handled properly #9897
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
added 2 commits
July 8, 2019 19:15
feerrenrut
reviewed
Jul 9, 2019
| raise ValueError("unknown tag name: %s"%tagName) | ||
|
|
||
| def _CharacterDataHandler(self,data): | ||
| def _CharacterDataHandler(self,data, processBufferedSurrogates=False): |
Contributor
There was a problem hiding this comment.
Why was the processBufferedSurrogates arg added?
Collaborator
Author
There was a problem hiding this comment.
Ah, I added this parameter without actually using it. Let me fix that.
|
For textUtils please see PR #9545. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Link to issue number:
None. I initially intended to fix this as part of the textUtils pr, but somehow it got lost in the process.
Summary of the issue:
In virtual buffers in the current threshold_py3_staging branch, emoji are shown as two surrogate characters instead of the actual emoji character.
This is because virtual buffers output in UTF-16, and surrogate characters are not allowed in xml. Therefore, the XML contains integer values for the surrogate characters, and they are converted to real characters in the xmlFormatting module. As surrogate characters are perfectly valid within Python 3 and even a surrogate pair is allowed, Python 3 does not collapse two surrogate characters into one 32-bit character.
Description of how this pull request fixes the issue:
If a low surrogate character is handled, add it to the currently buffered text, and quickly encode and decode the text to/from UTF-16. This ensures that surrogate pairs are properly decoded to the associated 32-bit character. The decision to do this for low surrogates only is intentional. A high surrogate is usually followed by a low surrogate. Therefore it doesn't make sense to re-encode if processing a high surrogate. We also want to avoid just re-encoding anything.
Testing performed:
Made sure that emoji read again in virtual buffers.
Known issues with pull request:
In xml full of emoji, encoding and decoding takes place multiple times. Yet, it makes more sense to me to do it this way than just reencoding every single piece of text.
Change log entry:
None