Skip to content

Commit

Permalink
Introduce IUIAutomation5 interface and ability to announce notificati…
Browse files Browse the repository at this point in the history
…ons in Windows 10 Fall Creators Update and later (#8045)

* UIA: support IUIAutomation5 so notification event tracking can be supported. Re #8009.

Also includes #7984: in Windows 10 Version 1709 (Fall Creators Update), UIA notification event is introduced for the benefit of screen readers so they can announct needed text. This is part of UIA 5 interface, which NVDA does not support yet.
The UIA notification event accepts additional parameters such as notification kind, notification processing constant and activit ID. This is useful for controlling how things should be announced and when. For now, the handler will do nothing.
Also, when initializing UIA handler, use UIA 5, and then fall back to UIA 4.

* UIA notification event: treat it the same as any other event except for additional kwargs for event execution. Re #8009.

Treat UIA notification event just like others except for two things:
* A new UIA_notification event will be fired.
* The new event will populate kwargs based on arguments for the actual event handling function (sender, displayString, etc.). This means app modules, global plugins, objects and others must include these kwargs when defining this event.

* UIA interface: obtain appropriate IUIAutomation interface based on Windows release. Re #8009.

A new function named getIUIAInterface (along with a private companion for Windows 10) is introduced to obtain highest supported IUIAutomation interface. Depending on Windows release, it'll return various interfaces. In case of Windows 10, a companion function will return appropriate interface based on build ranges.

* UIA notification event: add the handler if IUIAutomation5 is in use. Re #7984.

If IUIAutomation5 is in use, add notificaiton event handler, otherwise COM error is thrown.
Also, instead of relying solely on Python MRO to retrieve UIA interface, save the string somewhere for reference purposes and for future use.

* UIA notification event: camel case for consistency.

* NVDAObjects/UIA: handle UIA notification event from object level. Re #8009.

The base implementation for UIA notification handler will just speak whatever notification it receives. Subclasses are more than welcome to add further processing routines.

* UIA interface: query interface from version 5 downwards. Re #8009.

Comments from @LeonarddeR and @michaelDCurran: build ranges aren't useful, subsequently proven to be true. As an alternative, query interfaces from newest to oldest, which does allow NVDA to use IUIAutomation5 interface from source.
Also, this means getIUIAInterface function is no longer necessary.

* UIA support: check isinstance, no more IUIAVersion string. Re #8009.

Commented by @LeonarddeR and @michaelDCurran: check isinstance instead of keeping an eye on IUIA interface string.

* UIA notification: do not include sender argument. Re #8009.

Reviewed by @michaelDCurran (NV Access): sender argument is redundant.

* UIA notification: do not announce notifications from background apps. Re #8045.
  • Loading branch information
josephsl authored and michaelDCurran committed Mar 28, 2018
1 parent f6c88be commit 35c0840
Show file tree
Hide file tree
Showing 2 changed files with 33 additions and 5 deletions.
14 changes: 14 additions & 0 deletions source/NVDAObjects/UIA/__init__.py
Expand Up @@ -31,6 +31,7 @@
from NVDAObjects.behaviors import ProgressBar, EditableTextWithoutAutoSelectDetection, Dialog, Notification, EditableTextWithSuggestions
import braille
import time
import ui

class UIATextInfo(textInfos.TextInfo):

Expand Down Expand Up @@ -1399,6 +1400,19 @@ def event_UIA_systemAlert(self):
# Ideally, we wouldn't use getBrailleTextForProperties directly.
braille.handler.message(braille.getBrailleTextForProperties(name=self.name, role=self.role))

def event_UIA_notification(self, notificationKind=None, notificationProcessing=None, displayString=None, activityId=None):
"""
Introduced in Windows 10 Fall Creators Update (build 16299).
This base implementation announces all notifications from the UIA element.
Unlike other events, the text to be announced is not the name of the object, and parameters control how the incoming notification should be processed.
Subclasses can override this event and can react to notification processing instructions.
"""
# Do not announce notifications from background apps.
if self.appModule != api.getFocusObject().appModule:
return
if displayString:
ui.message(displayString)

class TreeviewItem(UIA):

def _get_value(self):
Expand Down
24 changes: 19 additions & 5 deletions source/_UIAHandler.py
Expand Up @@ -154,7 +154,7 @@
}

class UIAHandler(COMObject):
_com_interfaces_=[IUIAutomationEventHandler,IUIAutomationFocusChangedEventHandler,IUIAutomationPropertyChangedEventHandler]
_com_interfaces_=[IUIAutomationEventHandler,IUIAutomationFocusChangedEventHandler,IUIAutomationPropertyChangedEventHandler,IUIAutomationNotificationEventHandler]

def __init__(self):
super(UIAHandler,self).__init__()
Expand Down Expand Up @@ -187,10 +187,13 @@ def MTAThreadFunc(self):
except (COMError,WindowsError,NameError):
self.clientObject=CoCreateInstance(CUIAutomation._reg_clsid_,interface=IUIAutomation,clsctx=CLSCTX_INPROC_SERVER)
if isUIA8:
try:
self.clientObject=self.clientObject.QueryInterface(IUIAutomation3)
except COMError:
self.clientObject=self.clientObject.QueryInterface(IUIAutomation2)
# #8009: use appropriate interface based on highest supported interface.
for interface in (IUIAutomation5, IUIAutomation4, IUIAutomation3, IUIAutomation2):
try:
self.clientObject=self.clientObject.QueryInterface(interface)
break
except COMError:
pass
log.info("UIAutomation: %s"%self.clientObject.__class__.__mro__[1].__name__)
self.windowTreeWalker=self.clientObject.createTreeWalker(self.clientObject.CreateNotCondition(self.clientObject.CreatePropertyCondition(UIA_NativeWindowHandlePropertyId,0)))
self.windowCacheRequest=self.clientObject.CreateCacheRequest()
Expand All @@ -211,6 +214,9 @@ def MTAThreadFunc(self):
self.clientObject.AddPropertyChangedEventHandler(self.rootElement,TreeScope_Subtree,self.baseCacheRequest,self,UIAPropertyIdsToNVDAEventNames.keys())
for x in UIAEventIdsToNVDAEventNames.iterkeys():
self.clientObject.addAutomationEventHandler(x,self.rootElement,TreeScope_Subtree,self.baseCacheRequest,self)
# #7984: add support for notification event (IUIAutomation5, part of Windows 10 build 16299 and later).
if isinstance(self.clientObject, IUIAutomation5):
self.clientObject.AddNotificationEventHandler(self.rootElement,TreeScope_Subtree,self.baseCacheRequest,self)
except Exception as e:
self.MTAThreadInitException=e
finally:
Expand Down Expand Up @@ -293,6 +299,14 @@ def IUIAutomationPropertyChangedEventHandler_HandlePropertyChangedEvent(self,sen
obj=focus
eventHandler.queueEvent(NVDAEventName,obj)

def IUIAutomationNotificationEventHandler_HandleNotificationEvent(self,sender,NotificationKind,NotificationProcessing,displayString,activityId):
if not self.MTAThreadInitEvent.isSet():
# UIAHandler hasn't finished initialising yet, so just ignore this event.
return
import NVDAObjects.UIA
obj=NVDAObjects.UIA.UIA(UIAElement=sender)
eventHandler.queueEvent("UIA_notification",obj, notificationKind=NotificationKind, notificationProcessing=NotificationProcessing, displayString=displayString, activityId=activityId)

def _isUIAWindowHelper(self,hwnd):
# UIA in NVDA's process freezes in Windows 7 and below
processID=winUser.getWindowThreadProcessID(hwnd)[0]
Expand Down

0 comments on commit 35c0840

Please sign in to comment.