Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Python 3: When processing xml coming from virtual buffers, make sure that surrogate characters are handled properly #9897

merged 3 commits into from Jul 9, 2019


Copy link

@leonardder leonardder commented Jul 8, 2019

Link to issue number:

None. I initially intended to fix this as part of the textUtils pr, but somehow it got lost in the process.

Summary of the issue:

In virtual buffers in the current threshold_py3_staging branch, emoji are shown as two surrogate characters instead of the actual emoji character.
This is because virtual buffers output in UTF-16, and surrogate characters are not allowed in xml. Therefore, the XML contains integer values for the surrogate characters, and they are converted to real characters in the xmlFormatting module. As surrogate characters are perfectly valid within Python 3 and even a surrogate pair is allowed, Python 3 does not collapse two surrogate characters into one 32-bit character.

Description of how this pull request fixes the issue:

If a low surrogate character is handled, add it to the currently buffered text, and quickly encode and decode the text to/from UTF-16. This ensures that surrogate pairs are properly decoded to the associated 32-bit character. The decision to do this for low surrogates only is intentional. A high surrogate is usually followed by a low surrogate. Therefore it doesn't make sense to re-encode if processing a high surrogate. We also want to avoid just re-encoding anything.

Testing performed:

Made sure that emoji read again in virtual buffers.

Known issues with pull request:

In xml full of emoji, encoding and decoding takes place multiple times. Yet, it makes more sense to me to do it this way than just reencoding every single piece of text.

Change log entry:


@leonardder leonardder requested a review from feerrenrut Jul 8, 2019
@leonardder leonardder added this to In progress in Update NVDA to Python 3 via automation Jul 8, 2019
@@ -48,10 +55,12 @@ def _EndElementHandler(self,tagName):
raise ValueError("unknown tag name: %s"%tagName)

def _CharacterDataHandler(self,data):
def _CharacterDataHandler(self,data, processBufferedSurrogates=False):
Copy link

@feerrenrut feerrenrut Jul 9, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why was the processBufferedSurrogates arg added?

Copy link
Collaborator Author

@leonardder leonardder Jul 9, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, I added this parameter without actually using it. Let me fix that.

@leonardder leonardder requested a review from feerrenrut Jul 9, 2019
Update NVDA to Python 3 automation moved this from In progress to Reviewer approved Jul 9, 2019
Copy link

@feerrenrut feerrenrut left a comment

Thanks @leonardder

@feerrenrut feerrenrut merged commit f1c884d into nvaccess:threshold_py3_staging Jul 9, 2019
1 check passed
Update NVDA to Python 3 automation moved this from Reviewer approved to Done Jul 9, 2019
@nvaccessAuto nvaccessAuto added this to the 2019.3 milestone Jul 9, 2019
Copy link

@DrSooom DrSooom commented Jul 10, 2019

For textUtils please see PR #9545.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
None yet
No open projects
Linked issues

Successfully merging this pull request may close these issues.

None yet

4 participants