Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Python 3: When processing xml coming from virtual buffers, make sure that surrogate characters are handled properly #9897

merged 3 commits into from Jul 9, 2019


Copy link

leonardder commented Jul 8, 2019

Link to issue number:

None. I initially intended to fix this as part of the textUtils pr, but somehow it got lost in the process.

Summary of the issue:

In virtual buffers in the current threshold_py3_staging branch, emoji are shown as two surrogate characters instead of the actual emoji character.
This is because virtual buffers output in UTF-16, and surrogate characters are not allowed in xml. Therefore, the XML contains integer values for the surrogate characters, and they are converted to real characters in the xmlFormatting module. As surrogate characters are perfectly valid within Python 3 and even a surrogate pair is allowed, Python 3 does not collapse two surrogate characters into one 32-bit character.

Description of how this pull request fixes the issue:

If a low surrogate character is handled, add it to the currently buffered text, and quickly encode and decode the text to/from UTF-16. This ensures that surrogate pairs are properly decoded to the associated 32-bit character. The decision to do this for low surrogates only is intentional. A high surrogate is usually followed by a low surrogate. Therefore it doesn't make sense to re-encode if processing a high surrogate. We also want to avoid just re-encoding anything.

Testing performed:

Made sure that emoji read again in virtual buffers.

Known issues with pull request:

In xml full of emoji, encoding and decoding takes place multiple times. Yet, it makes more sense to me to do it this way than just reencoding every single piece of text.

Change log entry:


Leonard de Ruijter added 2 commits Jul 8, 2019
@leonardder leonardder requested a review from feerrenrut Jul 8, 2019
@leonardder leonardder added this to In progress in Update NVDA to Python 3 via automation Jul 8, 2019
@@ -48,10 +55,12 @@ def _EndElementHandler(self,tagName):
raise ValueError("unknown tag name: %s"%tagName)

def _CharacterDataHandler(self,data):
def _CharacterDataHandler(self,data, processBufferedSurrogates=False):

This comment has been minimized.

Copy link

feerrenrut Jul 9, 2019


Why was the processBufferedSurrogates arg added?

This comment has been minimized.

Copy link

leonardder Jul 9, 2019

Author Collaborator

Ah, I added this parameter without actually using it. Let me fix that.

@leonardder leonardder requested a review from feerrenrut Jul 9, 2019
Update NVDA to Python 3 automation moved this from In progress to Reviewer approved Jul 9, 2019
Copy link

feerrenrut left a comment

Thanks @leonardder

@feerrenrut feerrenrut merged commit f1c884d into nvaccess:threshold_py3_staging Jul 9, 2019
1 check passed
1 check passed
continuous-integration/appveyor/pr AppVeyor build succeeded
Update NVDA to Python 3 automation moved this from Reviewer approved to Done Jul 9, 2019
@nvaccessAuto nvaccessAuto added this to the 2019.3 milestone Jul 9, 2019

This comment has been minimized.

Copy link

DrSooom commented Jul 10, 2019

For textUtils please see PR #9545.

@josephsl josephsl referenced this pull request Jul 23, 2019
107 of 107 tasks complete
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
4 participants
You can’t perform that action at this time.