Punctuation read in symbol description when moving by character

[CyrilleB79]

The symbol description of the symbols in symbols.dic and cldr.dic is not read correctly when the description contains punctuation such as dash or apostrophe.
Below are two examples, one from each file.

Steps to reproduce:

• Ensure to have NVDA and TTS in English
• Read the two following symbols character by character moving the cursor with the arrows: ➔ 👕

Actual behavior:

➔ is read "right dash pointing arrow"
👕 is read "T dash shirt"

Expected behavior:

➔ should be read "right pointing arrow"
👕 should be read "T shirt"

System configuration

NVDA installed/portable/running from source:

NVDA version:

Last alpha
Version : source-master-0dbab4d

Windows version:

Windows 10 1809 (64-bit) build 17763.1935

Name and version of other software in use when reproducing the issue:

N/A

Other information about your system:

Other questions

Does the issue still occur after restarting your computer?

Not tested but surely yes.

Have you tried any other versions of NVDA? If so, please report their behaviors.

NVDA 2021.1 doesn't have this issue.

If add-ons are disabled, is your problem still occurring?

Yes

Does the issue still occur after you run the COM Registration Fixing Tool in NVDA's tools menu?

Not tested; but should not be related.

Notes

• Regression in alpha stage, probably introduced by Speak all symbols when moving by words (#11779) #11856. Cc @JulienCochuyt, @seanbudd, @feerrenrut
• The same issue also occurs in localized symbol and cldr files. With French translators, we have removed dashes and apostrophes in symbol descriptions of the symbol files. But then I have realized that the same issue was present in the CLDR file. Removing dashes, apostrophes and maybe other punctuation in symbol and CLDR files is probably not the solution. Cc @MichelSuch

[CyrilleB79]

And another much more important STR with French NVDA and TTS:

• Open Word (or go to whatever edit field where spelling mistakes are detected by NVDA)
• Type a sentence with a spelling mistake, e.g.: "Paris, Waashington, London".
• Review your text with left/right arrows

Actual
When you reach the spelling mistake, you hear "Faute d apostrophe orthographe" that stands for "spelling error".

Expected:
When you reach the spelling mistake, you should hear "Faute d'orthographe".

Asking translators to remove punctuations from their translated strings is clearly not the solution.

[seanbudd]

I realise that the 2020.4 behaviour would have been:

➔ should be read "right pointing arrow"
👕 should be read "T dash shirt"

But I believe the intended change of behaviour was to make this consistent. Maybe instead it should be:

➔ should be read "right pointing arrow"
👕 should be read "t shirt"

Reverting to the old behaviour is simple but doing the latter seems more tricky.

[CyrilleB79]

Sorry, the expected result in the initial description of this issue was erroneous. I have just updated it.

[seanbudd]

This seems to be caused by the text being processed twice, I raised an assertion within processText when certain text is read: '👕' for the first example and "t-shirt" for the second example.

First time stack trace

IO - speech.speech.speak (16:35:03.201) - MainThread (21088):
Speaking [LangChangeCommand ('en_US'), 'Text Editor', 'edit', 'multi line', CancellableSpeech (still valid), '👕']
ERROR - eventHandler.executeEvent (16:35:03.216) - MainThread (21088):
error executing event: gainFocus on <NVDAObjects.Dynamic_IAccessibleEditWindowNVDAObject object at 0x10985AB0> with extra args of {}
Traceback (most recent call last):
  File "eventHandler.py", line 246, in executeEvent
    _EventExecuter(eventName,obj,kwargs)
  File "eventHandler.py", line 96, in __init__
    self.next()
  File "eventHandler.py", line 105, in next
    return func(*args, **self.kwargs)
  File "NVDAObjects\behaviors.py", line 187, in event_gainFocus
    super().event_gainFocus()
  File "NVDAObjects\__init__.py", line 1137, in event_gainFocus
    self.reportFocus()
  File "NVDAObjects\__init__.py", line 1005, in reportFocus
    speech.speakObject(self, reason=controlTypes.OutputReason.FOCUS)
  File "speech\speech.py", line 542, in speakObject
    speak(sequence, priority=priority)
  File "speech\speech.py", line 842, in speak
    speechSequence[index]=processText(curLanguage,item,symbolLevel)
  File "speech\speech.py", line 131, in processText
    text = characterProcessing.processSpeechSymbols(locale, text, symbolLevel)
  File "characterProcessing.py", line 698, in processSpeechSymbols
    return ss.processText(text, level)

Second time stack trace

Speaking [LangChangeCommand ('en_US'), 't-shirt', EndUtteranceCommand()]
ERROR - scriptHandler.executeScript (16:37:16.593) - MainThread (1780):
error executing script: <bound method EditableText.script_caret_moveByCharacter of <NVDAObjects.Dynamic_IAccessibleEditWindowNVDAObject object at 0x141ACE70>> with gesture 'right arrow'
Traceback (most recent call last):
  File "scriptHandler.py", line 208, in executeScript
    script(gesture)
  File "editableText.py", line 228, in script_caret_moveByCharacter
    self._caretMovementScriptHelper(gesture, textInfos.UNIT_CHARACTER)
  File "editableText.py", line 163, in _caretMovementScriptHelper
    self._caretScriptPostMovedHelper(unit,gesture,newInfo)
  File "NVDAObjects\behaviors.py", line 177, in _caretScriptPostMovedHelper
    super()._caretScriptPostMovedHelper(speakUnit, gesture, info)
  File "editableText.py", line 149, in _caretScriptPostMovedHelper
    speech.speakTextInfo(info, unit=speakUnit, reason=controlTypes.OutputReason.CARET)
  File "speech\speech.py", line 1134, in speakTextInfo
    speak(seq, symbolLevel=symbolLevel, priority=priority)
  File "speech\speech.py", line 842, in speak
    speechSequence[index]=processText(curLanguage,item,symbolLevel)
  File "speech\speech.py", line 131, in processText
    text = characterProcessing.processSpeechSymbols(locale, text, symbolLevel)
  File "characterProcessing.py", line 698, in processSpeechSymbols
    return ss.processText(text, level)

[CyrilleB79]

I did not follow carefully the details of this PR.

However the following test shows that the hyphen should be passed to the synth:

• Using OneCore French (Hortense)
• The normal symbol prononciation written "T-shirt" is pronounced "tee shirt" (english reading) i.e. "ti cheurte" (French reading)
• Modifying the symbol prononciation to "T shirt" (without dash) in the symbol dialog leads to the following prononciation: "teh shirt" (English reading) i.e. "té cheurte" (French reading)
  HTH