Fast Diff Match Patch crashes and the calling thread deadlocks when astral characters are printed in Windows Terminal #16027
Description
Steps to reproduce:
- In the NVDA Advanced settings panel, insure that "Diff algorithm" is set to "Diff Match Patch" (default), and "Speak new text in Windows Terminal via" is set to "Diffing" (default).
- Run the Python interpreter in Windows Terminal.
- Execute the following Python print statement.
print("\N{SHORTCAKE}")
Actual behavior:
If running a recent build of NVDA that uses the Fast Diff Match Patch library:
- The nvda_dmp.exe process will throw a silent exception and terminate, disappearing from Task Manager. No log info is generated, even when NVDA is logging in debug mode.
- the thread that ran nvda_dmp.exe via subprocess will deadlock, because it is waiting for text which it will never receive.
- NVDA will no longer automatically speak new output in terminal windows, or any other window that inherits from NVDAObjects.behaviors.LiveText until restarted.
- NVDA will fail to play the exit sound when told to restart, because of the deadlocked thread.
Expected behavior:
if 'Include Unicode Consortium data is enabled in NVDA speech settings, the name of the emoji "Shortcake" should be spoken. If not, NVDA should not speak the name of the character. Either way, NVDA should be sanitizing the text before sending it to the nvda_dmp.exe process to prevent it from crashing, and to insure continued automatic speech output in terminal windows.
NVDA logs, crash dumps and other attachments:
System configuration
NVDA installed/portable/running from source:
installed
NVDA version:
2024.1beta2 (2024.1.0.30416)
Windows version:
Windows 11, version 23H2
Name and version of other software in use when reproducing the issue:
Windows Terminal, Version 1.18.3181.0
Other information about your system:
Not applicable.
Other questions
Does the issue still occur after restarting your computer?
yes if another astral character is printed to the terminal, or any window who's overlay class inherits from NVDAObjects.behaviors.LiveText.
Have you tried any other versions of NVDA? If so, please report their behaviors.
Bug is applicable to versions of NVDA that use Fast Diff Match Patch.
If NVDA add-ons are disabled, is your problem still occurring?
Not applicable.
Does the issue still occur after you run the COM Registration Fixing Tool in NVDA's tools menu?
Not applicable.
Some background
The Fast Diff Match Patch ReadMe says:
On Windows, an exception will be thrown if either of the two text strings has characters outside of the Basic Multilingual Plane because the native platform character type is a two-byte character.
Some Emoji characters (not all) fall outside of the Basic Multilingual Plane, in the so-called Astral Plane. Unfortunately, some console applications such as the Black formatter for Python output Emoji characters to the console, and in the case of Black, don't have the ability to disable this functionality.
When a recent build of NVDA which makes use of the new Fast Diff Match Patch library tries to automatically speak a character in the astral plane, the nvda_dmp.exe process terminates with an exception, and the NVDA thread that called nvda_dmp.exe is left blocking while it waits for the output of nvda_dmp.exe, which of course it will never receive. Because of the thread becoming deadlocked in this way, no log output is emitted, making the source of the issue tricky to track down.
I created an NVDA add-on that patches classes which inherit from NVDAObjects.behaviors.LiveText, in order to sanitize strings before Fast Diff Match Patch receives them and prevent nvda_dmp.exe from terminating unexpectedly. The add-on will speak astral character names if the user has enabled the 'Include Unicode Consortium data in the NVDA speech settings panel, otherwise it will omit speaking the character name, for consistency with how emoji and other unicode characters in the Basic Multilingual Plane are currently spoken. I've attached the add-on above, but a fix should really be integrated into NVDA its self, as this is an issue which affects NVDA's usage with at least one common CLI-based tool (I.E. Black) and probably several others.