Join GitHub today
GitHub is home to over 40 million developers working together to host and review code, manage projects, and build software together.Sign up
Python 3: When processing xml coming from virtual buffers, make sure that surrogate characters are handled properly #9897
Link to issue number:
None. I initially intended to fix this as part of the textUtils pr, but somehow it got lost in the process.
Summary of the issue:
In virtual buffers in the current threshold_py3_staging branch, emoji are shown as two surrogate characters instead of the actual emoji character.
Description of how this pull request fixes the issue:
If a low surrogate character is handled, add it to the currently buffered text, and quickly encode and decode the text to/from UTF-16. This ensures that surrogate pairs are properly decoded to the associated 32-bit character. The decision to do this for low surrogates only is intentional. A high surrogate is usually followed by a low surrogate. Therefore it doesn't make sense to re-encode if processing a high surrogate. We also want to avoid just re-encoding anything.
Made sure that emoji read again in virtual buffers.
Known issues with pull request:
In xml full of emoji, encoding and decoding takes place multiple times. Yet, it makes more sense to me to do it this way than just reencoding every single piece of text.
Change log entry: