Speak typed characters in UWP apps on Windows 10 RS2 and above #6631

michaelDCurran · 2016-12-08T07:00:50Z

This code will only work on Windows build 14986 and above.

for apps in Win 10 rs1 and above where we cannot inject in-process.

…ws.ui.Core windows from 14946 onwards.

…indows build 14986 and up.

jcsteh · 2016-12-16T02:41:02Z

source/keyboardHandler.py

@@ -162,6 +167,21 @@ def internal_keyDownEvent(vkCode,scanCode,extended,injected):
 				return False
 	except:
 		log.error("internal_keyDownEvent", exc_info=True)
+	finally:


Might be worth commenting why we do this in the finally block; i.e. because we return early in several places but we still want this to be executed.

jcsteh · 2016-12-16T02:42:09Z

source/keyboardHandler.py

+	finally:
+		# #6017: handle typed characters for UWP apps in Win10 RS2 and above where we can't detect typed characters in-process 
+		focus=api.getFocusObject()
+		if sys.getwindowsversion().build>=14986 and not gestureExecuted and not isNVDAModifierKey(vkCode,extended) and not vkCode in KeyboardInputGesture.NORMAL_MODIFIER_KEYS and focus.windowClassName.startswith('Windows.UI.Core'):


Calling sys.getwindowsversion probably isn't expensive, but we already have it cached in winVersion.winVersion anyway, so I'd use that instead.

jcsteh · 2016-12-16T02:43:41Z

source/keyboardHandler.py

+				keyStates[k]=ctypes.windll.user32.GetKeyState(k)
+			charBuf=ctypes.create_unicode_buffer(5)
+			# Normally calling ToUnicodeEx would destroy keyboard buffer state and therefore cause the app to not produce the right WM_CHAR message.
+			# However in UWP apps in Windows 10 RS2 and above, wm_keydown / wm_char is never processed, thus NVDA will be the only one calling it. 


Is this comment still true? I thought the point of the flag was to stop this state destroying behaviour, rather than it being something specific to UWP.

jcsteh · 2016-12-16T02:49:36Z

source/keyboardHandler.py

+			# Normally calling ToUnicodeEx would destroy keyboard buffer state and therefore cause the app to not produce the right WM_CHAR message.
+			# However in UWP apps in Windows 10 RS2 and above, wm_keydown / wm_char is never processed, thus NVDA will be the only one calling it. 
+			hkl=ctypes.windll.user32.GetKeyboardLayout(focus.windowThreadID)
+			res=ctypes.windll.user32.ToUnicodeEx(vkCode,scanCode,keyStates,charBuf,5,4,hkl)


I assume 4 is the flag? Please comment as to what the 4 does.

I assume 5 is the size? Perhaps use len(charBuf) to be clearer and so we don't break things if we change the size in future.

jcsteh · 2016-12-16T02:51:46Z

source/keyboardHandler.py

+			res=ctypes.windll.user32.ToUnicodeEx(vkCode,scanCode,keyStates,charBuf,5,4,hkl)
+			if res>0:
+				for ch in charBuf[:res]: 
+					eventHandler.queueEvent("typedCharacter",focus,ch=ch)


Is this saying there can be multiple separate characters? I know there are languages where a single key press produces a conjunct Unicode character sequence (e.g. Tamil), but we probably actually want that to be handled as one character. We can't do that with WM_CHAR right now, but is there a reason we shouldn't do this here since we can?

I have certainly seen it produce at least 2 characters yes. For instance, if a dead key is hit, followed by a character that is not compatible with a dead key, then you get a char for the dead key (say ^) and also the actual character.

michaelDCurran · 2016-12-16T03:12:56Z

Actually, technically the flag is only needed for non-uwp apps as TranslateMessage is never called. So technically, we could just call it with a flag of 0 and that comment would be correct. However, if we were to ever extend the use of this code to outside of UWP apps, then the flag is most certainly needed. The actual mS bug I filed was to ensure that the flag produced the same results whether it was UWP or not. As in older Windows builds, the flag in UWP apps breaks things just as badly as if ToUnicodeEx were used in conjunction with TranslateMessage, and no flag given.

Nikita34196 · 2016-12-24T19:06:52Z

When will this fix in nvda next? and will be in the release 2017.1

derekriemer · 2016-12-25T03:10:05Z

It only will work for insider builds until rs2. Also @michaelDCurran I'd recommend this incubate in next for longer than usual since it won't be tested until RS2.

…

On 12/24/2016 12:06 PM, nikita wrote: When will this fix in nvda next? and will be in the release 2017.1 — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#6631 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFGivUu8sHfY0OVlffat6EYk9kd9xwKqks5rLW1NgaJpZM4LHgGc>.

-- ------------------------------------------------------------------------ Derek Riemer * Department of computer science, third year undergraduate student. * Proud user of the NVDA screen reader. * Open source enthusiast. * Member of Bridge Cu * Avid skiier. Websites: Honors portfolio <http://derekriemer.com> Awesome little hand built weather app! <http://django.derekriemer.com/weather/> email me at derek.riemer@colorado.edu <mailto:derek.riemer@colorado.edu> Phone: (303) 906-2194

josephsl · 2016-12-25T04:54:11Z

Hi, RS2 is under development, so I do second Derek’s proposal that this get incubated for a long time (at least three weeks, I’d say). Thanks. From: derekriemer [mailto:notifications@github.com] Sent: Saturday, December 24, 2016 7:10 PM To: nvaccess/nvda <nvda@noreply.github.com> Cc: Subscribed <subscribed@noreply.github.com> Subject: Re: [nvaccess/nvda] Speak typed characters in UWP apps on Windows 10 RS2 and above (#6631) It only will work for insider builds until rs2. Also @michaelDCurran I'd recommend this incubate in next for longer than usual since it won't be tested until RS2.

…

On 12/24/2016 12:06 PM, nikita wrote: When will this fix in nvda next? and will be in the release 2017.1 — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#6631 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFGivUu8sHfY0OVlffat6EYk9kd9xwKqks5rLW1NgaJpZM4LHgGc>.

-- ------------------------------------------------------------------------ Derek Riemer * Department of computer science, third year undergraduate student. * Proud user of the NVDA screen reader. * Open source enthusiast. * Member of Bridge Cu * Avid skiier. Websites: Honors portfolio <http://derekriemer.com> Awesome little hand built weather app! <http://django.derekriemer.com/weather/> email me at derek.riemer@colorado.edu <mailto:derek.riemer@colorado.edu> Phone: (303) 906-2194 — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#6631 (comment)> , or mute the thread <https://github.com/notifications/unsubscribe-auth/AHgLkAUFLJRN0g6x8BLZxW_FWlLCweEMks5rLd6OgaJpZM4LHgGc> .

michaelDCurran · 2017-01-06T02:27:27Z

@jcsteh: I have made all requested changes.

josephsl · 2017-01-09T03:56:48Z

Hi,

Critical bug found: After installing next.13789, try typing into Start menu or UWP apps. At least in build 14986 (my laptop), I get a traceback that says:

ERROR - unhandled exception (19:55:28):
Traceback (most recent call last):
File "_ctypes/callbacks.c", line 315, in 'calling callback function'
File "winInputHook.pyc", line 45, in keyboardHook
File "keyboardHandler.pyc", line 183, in internal_keyDownEvent
WindowsError: exception: access violation writing 0x00000005

Same result when I try to type an address into Edge. This was with speak typed chars set to off. Thanks.

michaelDCurran · 2017-01-09T03:59:09Z

Oops. My mistake. That function is missing an argument.

josephsl · 2017-01-21T23:57:56Z

Hi,

Try the following:

From build 15014 running on a laptop, go to Settings/Network/Status.
Open Show Available networks.
Press TAB to go through the network flyout until you arrive at the list of networks.
Select one, then press TAB to go to either Properties or Connect/Disconnect.
Press Escape.

Expected: no errors.
Actual:

josephsl · 2017-01-22T00:01:14Z

Hi,

Try the following:

On a laptop running build 15014, go to Settings/Network/Status.
Go to Show Available networks.
Select a wireless network and tab to either Properties or connect/disconnect.
Press Escape.

Expected: no errors.
Actual:

ERROR - eventHandler.executeEvent (15:47:46):
error executing event: typedCharacter on <NVDAObjects.UIA.UIA object at 0x050A8ED0> with extra args of {'ch': u'\x1b'}
Traceback (most recent call last):
File "eventHandler.pyc", line 143, in executeEvent
File "eventHandler.pyc", line 91, in init
File "eventHandler.pyc", line 98, in next
File "NVDAObjects_init_.pyc", line 840, in event_typedCharacter
File "speech.pyc", line 627, in speakTypedCharacters
File "api.pyc", line 235, in isTypingProtected
File "baseObject.pyc", line 34, in get
File "baseObject.pyc", line 110, in getPropertyViaCache
File "NVDAObjects\UIA_init.pyc", line 801, in _get_states
File "baseObject.pyc", line 34, in get
File "baseObject.pyc", line 110, in getPropertyViaCache
File "NVDAObjects\UIA_init.pyc", line 797, in _get_UIACachedStatesElement
COMError: (-2147467259, 'Unspecified error', (None, None, None, 0, None))

This happens on next.13789 and later, thus I suspect this has to do with this pull request.

Thanks.

michaelDCurran · 2017-01-23T00:47:10Z

@josephsl I'd say that if speak typed characters had originally worked in UWP apps without this code, we'd see the same errors here. As they don't impact on usage, I won't hold back this PR. However we may consider cleaning up the way errors in api.isprotected are handled, as this does seem to happen quite a bit.

Michael Curran and others added 4 commits October 5, 2016 10:56

keyboardHandler.internal_keyDownEvent: report tyed characters manually,

ad5cfc9

for apps in Win 10 rs1 and above where we cannot inject in-process.

keyboardHandler: use ToUnicodeEx to report typed characters for Windo…

19e5352

…ws.ui.Core windows from 14946 onwards.

Merge branch 'master' into i6017

41e23ec

keyboardHandler: handle typed characters with ToUnicodeEx only from W…

73fd374

…indows build 14986 and up.

michaelDCurran requested a review from jcsteh December 8, 2016 07:01

jcsteh requested changes Dec 16, 2016

View reviewed changes

Address review for #6631

e1a6ef7

jcsteh approved these changes Jan 9, 2017

View reviewed changes

michaelDCurran added a commit that referenced this pull request Jan 9, 2017

Incubates #6631

3439000

nvaccessAuto added the incubating label Jan 9, 2017

nvaccessAuto assigned michaelDCurran Jan 9, 2017

Add missing function argument! #6631

0a7b024

michaelDCurran added a commit that referenced this pull request Jan 9, 2017

Incubates #6631

2ffe5c4

michaelDCurran merged commit 6406413 into master Jan 23, 2017

nvaccessAuto removed the incubating label Jan 23, 2017

nvaccessAuto added this to the 2017.1 milestone Jan 23, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Speak typed characters in UWP apps on Windows 10 RS2 and above #6631

Speak typed characters in UWP apps on Windows 10 RS2 and above #6631

michaelDCurran commented Dec 8, 2016

jcsteh Dec 16, 2016

jcsteh Dec 16, 2016

jcsteh Dec 16, 2016

jcsteh Dec 16, 2016

jcsteh Dec 16, 2016

michaelDCurran Dec 16, 2016

michaelDCurran commented Dec 16, 2016

Nikita34196 commented Dec 24, 2016

derekriemer commented Dec 25, 2016 via email

josephsl commented Dec 25, 2016 via email

michaelDCurran commented Jan 6, 2017

josephsl commented Jan 9, 2017

michaelDCurran commented Jan 9, 2017 via email

josephsl commented Jan 21, 2017

josephsl commented Jan 22, 2017

michaelDCurran commented Jan 23, 2017

Speak typed characters in UWP apps on Windows 10 RS2 and above #6631

Speak typed characters in UWP apps on Windows 10 RS2 and above #6631

Conversation

michaelDCurran commented Dec 8, 2016

jcsteh Dec 16, 2016

Choose a reason for hiding this comment

jcsteh Dec 16, 2016

Choose a reason for hiding this comment

jcsteh Dec 16, 2016

Choose a reason for hiding this comment

jcsteh Dec 16, 2016

Choose a reason for hiding this comment

jcsteh Dec 16, 2016

Choose a reason for hiding this comment

michaelDCurran Dec 16, 2016

Choose a reason for hiding this comment

michaelDCurran commented Dec 16, 2016

Nikita34196 commented Dec 24, 2016

derekriemer commented Dec 25, 2016 via email

josephsl commented Dec 25, 2016 via email

michaelDCurran commented Jan 6, 2017

josephsl commented Jan 9, 2017

michaelDCurran commented Jan 9, 2017 via email

josephsl commented Jan 21, 2017

josephsl commented Jan 22, 2017

michaelDCurran commented Jan 23, 2017