Introduce IUIAutomation5 interface and ability to announce notifications in Windows 10 Fall Creators Update and later #8045

josephsl · 2018-02-28T01:18:11Z

Link to issue number:

Summary of the issue:

Introduce IUIAutomation5 interface and ability to announce notifications in Windows 10 Fall Creators Update an later

Description of how this pull request fixes the issue:

In Windows 10 Fall Creators Update and later, apps can inform screen readers to announce specific notification text. This is done via IUIAutomationNotificationEventHandler::HandleNotificationEvent, which is part of IUIAutomation5 interface. Since NVDA only uses IUIAutomation3, update UIA subsystem to use IUIAutomation5. Also, introduce ability to use more recent IUIAutomation interfaces on various Windows 10 releases (such as IUIAutomation4 in Anniversary Update/build 14393).

Testing performed:

Testing done via Windows 10 App Essentials add-on: using recent versions of various app (such as Calculator, Microsoft Store and others) that uses this new event to announce texts.

Known issues with pull request:

None so far.

Change log entry:

New feature: In Windows 10 Fall Creators Update and later, NVDA can announce notifications from apps such as Calculator and Windows Store.

…ported. Re nvaccess#8009. Also includes nvaccess#7984: in Windows 10 Version 1709 (Fall Creators Update), UIA notification event is introduced for the benefit of screen readers so they can announct needed text. This is part of UIA 5 interface, which NVDA does not support yet. The UIA notification event accepts additional parameters such as notification kind, notification processing constant and activit ID. This is useful for controlling how things should be announced and when. For now, the handler will do nothing. Also, when initializing UIA handler, use UIA 5, and then fall back to UIA 4.

…or additional kwargs for event execution. Re nvaccess#8009. Treat UIA notification event just like others except for two things: * A new UIA_notification event will be fired. * The new event will populate kwargs based on arguments for the actual event handling function (sender, displayString, etc.). This means app modules, global plugins, objects and others must include these kwargs when defining this event.

…ndows release. Re nvaccess#8009. A new function named getIUIAInterface (along with a private companion for Windows 10) is introduced to obtain highest supported IUIAutomation interface. Depending on Windows release, it'll return various interfaces. In case of Windows 10, a companion function will return appropriate interface based on build ranges.

nvaccess#7984. If IUIAutomation5 is in use, add notificaiton event handler, otherwise COM error is thrown. Also, instead of relying solely on Python MRO to retrieve UIA interface, save the string somewhere for reference purposes and for future use.

…vaccess#8009. The base implementation for UIA notification handler will just speak whatever notification it receives. Subclasses are more than welcome to add further processing routines.

LeonarddeR · 2018-02-28T05:06:36Z

Hey Joseph,

It seems you defined the getIUIAInterface function, but you never use it in your code below.
As noted on Gitter, I'm honestly not a fan of the build version checking, as it does not work when running from source. Furthermore, I'm not convinced of whether it is significantly faster than just querying interfaces from 5 up to 2. Some timing on my end revealed that querying interfaces is actually very fast.

michaelDCurran · 2018-02-28T05:13:29Z

Querying interfaces on the CUIAutomation client object should be very fast as the UI automation client COM interfaces run entirely within NVDA's process. Also, this will only be done once, on NVDA start anyway I'm assuming.

michaelDCurran · 2018-02-28T06:15:31Z

Do you know if this causes any double speaking where Microsoft may have fired both liveRegionChange and notification for the same thing?

josephsl · 2018-02-28T09:30:50Z

Hi, yes, but apps I came across so far raises one of these at runtime, or prefers the newer events. For example, I expected Microsoft Store app to fire either live region change or notification event in Creators Update (Version 1703), but it raised only notification event in that version. But I would imagine third-party apps may raise both events (perhaps due to oversight), but I guess preparing for that might be in order somehow. Thanks. From: Michael Curran [mailto:notifications@github.com] Sent: Tuesday, February 27, 2018 10:16 PM To: nvaccess/nvda <nvda@noreply.github.com> Cc: Joseph Lee <joseph.lee22590@gmail.com>; Author <author@noreply.github.com> Subject: Re: [nvaccess/nvda] Introduce IUIAutomation5 interface and ability to announce notifications in Windows 10 Fall Creators Update and later (#8045) Do you know if this causes any double speaking where Microsoft may have fired both liveRegionChange and notification for the same thing? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#8045 (comment)> , or mute the thread <https://github.com/notifications/unsubscribe-auth/AHgLkPHs4jSUrVFkBixQQAr01tdROGpqks5tZO8EgaJpZM4SV6jq> .

josephsl · 2018-02-28T09:40:59Z

Hi,

@LeonarddeR: the Win10 version is the one to be used from binary builds and eventually from source once we move to Python 3. The generic version is meant for add-ons and to show that NVDA still supports Windows 7 SP1. The reason for this design is to keep it compatible with what we have at the moment: falling back to IUIAutomation2 if newer versions are not available. As @michaelDCurran pointed out, this is meant to be used at startup (both for Core and add-ons).

Thanks.

michaelDCurran · 2018-02-28T09:59:43Z

As all IUIAutomationX interfaces inherit from the one before, there is no issue with compatibility. I.e. code written for windows 7 can interact with an IUIAutomation5 object just as it was an IUIAutomation2 object. At start-up, NVDA should simply QueryInterface from IUIAutomation5 down to IUIAutomation until one succeeds. Thus UIAHandler.handler.clientObject will always be the highest one the Operating System happens to support.

josephsl · 2018-02-28T11:47:21Z

Hi,

In the end, the getIUIAInterface function won't be added, as a for loop will be used to try various versions (modifying specs accordingly).

Thanks.

@LeonarddeR

…8009. Comments from @LeonarddeR and @michaelDCurran: build ranges aren't useful, subsequently proven to be true. As an alternative, query interfaces from newest to oldest, which does allow NVDA to use IUIAutomation5 interface from source. Also, this means getIUIAInterface function is no longer necessary.

LeonarddeR

I know you didn't ask for my review, but I'm trying to update my UIA knowledge, so that's why I looked into this.

LeonarddeR · 2018-02-28T18:18:22Z

source/NVDAObjects/UIA/__init__.py

@@ -1368,6 +1369,16 @@ def event_UIA_systemAlert(self):
 		# Ideally, we wouldn't use getBrailleTextForProperties directly.
 		braille.handler.message(braille.getBrailleTextForProperties(name=self.name, role=self.role))

+	def event_UIA_notification(self, sender=None, notificationKind=None, notificationProcessing=None, displayString=None, activityId=None):


I think you could leave out the sender kwarg. See my comment below.

LeonarddeR · 2018-02-28T18:20:29Z

source/_UIAHandler.py

+			return
+		import NVDAObjects.UIA
+		obj=NVDAObjects.UIA.UIA(UIAElement=sender)
+		eventHandler.queueEvent("UIA_notification",obj, sender=sender, notificationKind=NotificationKind, notificationProcessing=NotificationProcessing, displayString=displayString, activityId=activityId)


As the sender is equal to obj.UIAElement, I think it is a redundant kwarg. For consistency reasons, if one whould like to access the UIAElement for this event within a subclass, people can use the UIAElement property on the object.

I agree with @LeonarddeR here. Sender should not be explicitly provided to the notification event as it is redundant.

LeonarddeR · 2018-02-28T18:21:35Z

source/_UIAHandler.py

-					self.clientObject=self.clientObject.QueryInterface(IUIAutomation2)
-			log.info("UIAutomation: %s"%self.clientObject.__class__.__mro__[1].__name__)
+				# #8009: use appropriate interface based on highest supported interface.
+				for interface in (IUIAutomation5, IUIAutomation4, IUIAutomation3, IUIAutomation2):


May be put these interfaces in a constant?

LeonarddeR · 2018-02-28T18:26:20Z

source/_UIAHandler.py

+					except COMError:
+						pass
+			# Have this handy because we want to add things corresponding to the interface version.
+			IUIAVersion = self.clientObject.__class__.__mro__[1].__name__


Now you loop through these interfaces (which I think is a good thing to do), you might as well do something like this, avoiding digging into the mro:

for interface in interfaces: QueryInterface except: pass # Or continue? I guess they are identical in result here else: IUIAVersion = interface.__name__ ...

Since you just left the loop, you can even leave out the else statement, but using an else for the try statement looks somewhat cleaner to me. It's a matter of preference, though.

UiaVersion would also have to be set to IUIAutomation when isUIA8 is false. if you do it the way @LeonarddeR suggests

LeonarddeR · 2018-02-28T18:44:42Z

source/_UIAHandler.py

@@ -206,6 +211,9 @@ def MTAThreadFunc(self):
 			self.clientObject.AddPropertyChangedEventHandler(self.rootElement,TreeScope_Subtree,self.baseCacheRequest,self,UIAPropertyIdsToNVDAEventNames.keys())
 			for x in UIAEventIdsToNVDAEventNames.iterkeys():  
 				self.clientObject.addAutomationEventHandler(x,self.rootElement,TreeScope_Subtree,self.baseCacheRequest,self)
+			# #7984: add support for notification event (IUIAutomation5, part of Windows 10 build 16299 and later).
+			if IUIAVersion >= "IUIAutomation5":


Hmm, even though Python uses lexicographical comparison for strings and therefore this is code that does the write thing, I'd feel safer if you'd avoid this. As @michaelDCurran pointed out in #8045 (comment), all UIA interfaces inherit from the one before, so you could do an instance check on the client object for IUIAutomation5.

michaelDCurran · 2018-02-28T21:43:52Z

source/_UIAHandler.py

+						self.clientObject=self.clientObject.QueryInterface(interface)
+						break
+					except COMError:
+						pass


Perhaps an else on the for loop that raises RuntimeError("No UIAutomation client interface supported")

michaelDCurran · 2018-02-28T21:50:18Z

source/_UIAHandler.py

+					except COMError:
+						pass
+			# Have this handy because we want to add things corresponding to the interface version.
+			IUIAVersion = self.clientObject.__class__.__mro__[1].__name__


UiaVersion would also have to be set to IUIAutomation when isUIA8 is false. if you do it the way @LeonarddeR suggests

@LeonarddeR

…s#8009. Commented by @LeonarddeR and @michaelDCurran: check isinstance instead of keeping an eye on IUIA interface string.

josephsl · 2018-03-01T17:10:35Z

Hi,

In the end, isinstance checking will be done. I don't think an "else" is necessary except if we need to break out of the loop with that method.

Additional testing done by running the launcher (based on source code commits) on a Windows 8.1 system, and things work as expected. Thanks.

Brian1Gaff · 2018-03-01T18:03:22Z

Does this mean that if Microsoft bring out an even higher number this will need to be added or are they now finished twiddling about with the versions?

josephsl · 2018-03-01T18:07:31Z

Hi,

If the new interface includes useful features that screen reader users need to see/hear, then yes, we need to add support for it.

Thanks.

michaelDCurran · 2018-03-01T19:03:01Z

source/_UIAHandler.py

+			return
+		import NVDAObjects.UIA
+		obj=NVDAObjects.UIA.UIA(UIAElement=sender)
+		eventHandler.queueEvent("UIA_notification",obj, sender=sender, notificationKind=NotificationKind, notificationProcessing=NotificationProcessing, displayString=displayString, activityId=activityId)


I agree with @LeonarddeR here. Sender should not be explicitly provided to the notification event as it is redundant.

@michaelDCurran

Reviewed by @michaelDCurran (NV Access): sender argument is redundant.

# Conflicts: # source/NVDAObjects/UIA/__init__.py # source/_UIAHandler.py

… Re nvaccess#8045.

josephsl added 6 commits February 23, 2018 02:13

UIA notification event: camel case for consistency.

babcef8

NVDAObjects/UIA: handle UIA notification event from object level. Re n…

14a1074

…vaccess#8009. The base implementation for UIA notification handler will just speak whatever notification it receives. Subclasses are more than welcome to add further processing routines.

josephsl requested a review from michaelDCurran February 28, 2018 01:18

josephsl mentioned this pull request Feb 28, 2018

UIA support: choose the highest IUIAutomation interface version based on Windows release in use #8009

Closed

LeonarddeR reviewed Feb 28, 2018

View reviewed changes

michaelDCurran reviewed Feb 28, 2018

View reviewed changes

josephsl added 2 commits March 1, 2018 08:25

Merge branch 'master' into iuiautomation5

c3840db

UIA support: check isinstance, no more IUIAVersion string. Re nvacces…

b244561

…s#8009. Commented by @LeonarddeR and @michaelDCurran: check isinstance instead of keeping an eye on IUIA interface string.

michaelDCurran requested changes Mar 1, 2018

View reviewed changes

UIA notification: do not include sender argument. Re nvaccess#8009.

1889803

Reviewed by @michaelDCurran (NV Access): sender argument is redundant.

michaelDCurran approved these changes Mar 5, 2018

View reviewed changes

michaelDCurran added a commit that referenced this pull request Mar 5, 2018

Incubates #8045

5664089

nvaccessAuto added the incubating label Mar 5, 2018

nvaccessAuto assigned michaelDCurran Mar 5, 2018

Merge branch 'master' into iuiautomation5

d9cdbe9

# Conflicts: # source/NVDAObjects/UIA/__init__.py # source/_UIAHandler.py

josephsl added 2 commits March 9, 2018 17:36

Merge branch 'master' into iuiautomation5

4334782

UIA notification: do not announce notifications from background apps.…

29ab60d

… Re nvaccess#8045.

michaelDCurran approved these changes Mar 11, 2018

View reviewed changes

michaelDCurran added a commit that referenced this pull request Mar 11, 2018

Incubates #8045

43339f5

michaelDCurran merged commit 35c0840 into nvaccess:master Mar 28, 2018

nvaccessAuto removed the incubating label Mar 28, 2018

nvaccessAuto added this to the 2018.2 milestone Mar 28, 2018

josephsl deleted the iuiautomation5 branch September 16, 2019 02:05

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Introduce IUIAutomation5 interface and ability to announce notifications in Windows 10 Fall Creators Update and later #8045

Introduce IUIAutomation5 interface and ability to announce notifications in Windows 10 Fall Creators Update and later #8045

josephsl commented Feb 28, 2018 •

edited

LeonarddeR commented Feb 28, 2018

michaelDCurran commented Feb 28, 2018 via email

michaelDCurran commented Feb 28, 2018

josephsl commented Feb 28, 2018 via email

josephsl commented Feb 28, 2018

michaelDCurran commented Feb 28, 2018 via email

josephsl commented Feb 28, 2018

LeonarddeR left a comment

LeonarddeR Feb 28, 2018

LeonarddeR Feb 28, 2018

michaelDCurran Mar 1, 2018

LeonarddeR Feb 28, 2018

LeonarddeR Feb 28, 2018 •

edited

michaelDCurran Feb 28, 2018

LeonarddeR Feb 28, 2018

michaelDCurran Feb 28, 2018

michaelDCurran Feb 28, 2018

josephsl commented Mar 1, 2018

Brian1Gaff commented Mar 1, 2018 via email

josephsl commented Mar 1, 2018

michaelDCurran Mar 1, 2018

Introduce IUIAutomation5 interface and ability to announce notifications in Windows 10 Fall Creators Update and later #8045

Introduce IUIAutomation5 interface and ability to announce notifications in Windows 10 Fall Creators Update and later #8045

Conversation

josephsl commented Feb 28, 2018 • edited

Link to issue number:

Summary of the issue:

Description of how this pull request fixes the issue:

Testing performed:

Known issues with pull request:

Change log entry:

LeonarddeR commented Feb 28, 2018

michaelDCurran commented Feb 28, 2018 via email

michaelDCurran commented Feb 28, 2018

josephsl commented Feb 28, 2018 via email

josephsl commented Feb 28, 2018

michaelDCurran commented Feb 28, 2018 via email

josephsl commented Feb 28, 2018

LeonarddeR left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

LeonarddeR Feb 28, 2018 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

josephsl commented Mar 1, 2018

Brian1Gaff commented Mar 1, 2018 via email

josephsl commented Mar 1, 2018

Choose a reason for hiding this comment

josephsl commented Feb 28, 2018 •

edited

LeonarddeR Feb 28, 2018 •

edited