Windows 10/11 Calculator: allow NVDA to announce more operations by suppressing only a limited number of keyboard commands

[josephsl]

Hi,

This amounts to a substantial redesign of Windows 10/11 Calculator app module, therefore I think it would be best to queue this for 2022.2 or later:

Background:

A consequence of Windows as a Service (WaaS) introduced with Windows 10 (2015) is continuous ap updates, specifically apps preinstalled on PC's. One of these is Calculator (calculator.exe), which went through different user interface designs and added features over the years.

One of the biggest changes in Calculator came in 2018. With Windows 10 Version 1709 (Fal Creators Update) adding UIA notification event (part of IUIAutomation5 interface), Calculator app uses notification event to announce calculator display content changes. Prior to this, a combination of name change and live region was used to announce display content. With NVDA taking advantage of UIA notification event in 2018, it became possible for the screen reader to announce display content on supported Calculator app releases.

Then in 2020, NVDA added support for Windows 10 Calculator, which originated from Windows App Essentials add-on. In its early days, the NVDA Core version of Calculator app allowed NVDA to announce display content when a limited number of keyboard keys are pressed such as Escape, Enter, and Delete. Refinements followed such as announcing display content on braille displays, and with 2022.1, Windows 11 Calculator support.

In its original design, Calculator app module does not announce display content when number row keys are pressed. In fact, there are keyboard shortcuts in scientific mode such as "S" for sine that are not announced by NVDA. This stems from the fact that NVDA ignores DisplayUpdated activity Id altogether in order to solve double announcement problem where" speak typed characters" enabled causes NVDA to announce both the entered number and display content (for example, typing "2" results in NVDA saying "2, display is 2"). This design led to a discussion on NVDA Users list (see the reference in the end) where a user brought up the following suggestion.

Is your feature request related to a problem? Please describe.

When a calculator command such as + (plus) or = (equals) is pressed, NVDA does not announce results (specifically, calculator display content). Further, scientific mode results are not announced when operations such as trigonometric, logarithmic, factorial and other functions are performed. The issue was traced to the original design of Calculator app module (see above).

Describe the solution you'd like

Redesign the UIA notification event handler so it can ignore commands that should not result in NVDA announcing display content. For now number row keys will be marked as "no calculator results" list. Also, change calculator results script so it can just send gestures when using Windows 10 Version 1709 or later as this script was really optimized for an era without UIA notification event (virtually prior to 2018). See the "additional context" section on justifications.

Specifically:

• Current calculator results script wil be edited to just send the gesture if using Windows 10 Version 1709 or later.
• A new script (tentatively named "noCalculatorResults") will e defined and assigned to number row keys (1 through 0). The job of this script is to enable a flag that will inform UIA notification event handler to suppress announcement of display content.
• A private flag will be set up when number keys are entered and cleared inside the UIA notification event handler.
• UIA notification event handler will be edited to return if no results flag is enabled.

Describe alternatives you've considered

Define a list of keyboard shortcuts for Calculator that NVDA should announce. Note that this approach is not really future-proof because Microsoft can add or change commands in the future, made a bit easier to predict since Calculator is open-source (hosted on GitHub). Another option is keep Calculator app module in Core as is and let add-ons change the behavior, but that is not ideal for folks who may not elect to use add-ons for any reason (and in case NVDA is downloaded from Microsoft Store on educational devices running Windows 10 in S mode or Windows 11 SE).

Additional context

Below highlights possible questions and justifications for the proposed solution.

Why UIA notification event?

Increasingly, as discussed in a recent Windows Terminal issue that @codeofdusk commented, Microsoft is electing to use UIA notification event (prototype in Terminal) to notify assistive technologies about screen content changes. The best example, although not working properly for now, is Voice Access in Windows 11 Insider Preview where UIA notification event is used to communicate microphone status; a working example is page load status in Microsoft Edge.

Calculator is no exception, more so since 2018. Calculator does use UIA notification event to communicate screen content changes. Since the issue at hand is NVDA's UIA notification event handler (from client side), it makes sense to refine Calculator's own notification event handler method.

You mention Windows 10 Version 1709 a lot. Why?

This is the feature update that introduced UIA notification event, and from practical point of view, it is the earliest Windows 10 release that the proposed solution is applicable to. However, as explained in the next justification, 1709 will not make the cut.

How about older Windows 10 releases?

Older Windows 10 releases are not supported by Microsoft except for mission-critical devices. Since these devices are really meant for specialized uses, a version of Windows 10 Enterprise, aprtly named "Windows 10 Enterprise Long-Term Servicing Branch/Channel" is available. Windows 10 Enterprise LTSB/LTSC does not include modern apps such as Calculator, making the proposed solution not applicable.

But wait, you said Version 1709 did not make the cut. Why?

As of time of this issue submission, the oldest supported Windows 10 release (apart from LTSB/LTSC releases) is Version 1909 (November 2019 Update) which is going out of support in May 2022. Therefore, from practical point of view, the earliest version that the proposed solution is applicable is Version 20H2 (October 2020 Update). The latest version of Calculator for Windows 10 is 10.2103.8.0, which supports Version 1803 (April 2018 Update) or later.

Will the proposed solution work on Windows 11?

Yes, as Windows 10 and 11 Calculator support comes from the base Calculator app module.

Is there a prototype?

Yes - Windows App Essentials February 23rd development snapshot or later includes a prototype of the proposed solution.

Why delay the implementation to 2022.2 or later?

NVDA 2022.1 beta is very close at hand - master to beta and beta to master data exchange took place a few days ago, which is a key sign that signifies the imminent release of a beta version for wider testing. Also, since 2022.1 is a compatibility breaking release, more effort should be spent on testing these changes. Also, the proposed solution is in is infancy (prototype just released), it will take weeks to gather additional data, which can then inform the accompanying pull request.

References:

• Windows Calculator repository: https://github.com/microsoft/calculator
• UIA notification event (Win32): https://microsoft.github.io/windows-docs-rs/doc/windows/Win32/UI/Accessibility/fn.UiaRaiseNotificationEvent.html
• NVDA Users list thread on Windows Calculator issue described: https://nvda.groups.io/g/nvda/topic/problem_with_windows/88946458?p=,,,20,0,0,0::recentpostdate/sticky,,,20,2,0,88946458,previd=1645603799979387364,nextid=1645412580606544441&previd=1645603799979387364&nextid=1645412580606544441

Thanks.

[seanbudd]

Thank you @josephsl for all these details.

Could the technical, development and implementation details be moved to an empty (zero-diff) PR, i.e a new branch off of master?
This is so that we can focus the issue itself on the user impact.

[josephsl]

Hi, that’s the plan once I get around to doing this work, perhaps as early as late northern spring (May) as with all of my other past PR’s. Thanks.

[josephsl]

Hi,

March 2022 update: as noted in the additional context section, 1709 will not make the cut - the latest Calculator release for Windows 10 Fall Creators Update (unsupported) is 10.1804 which apparently does not use UIA notification event. While it would be better to check Calculator version (10.1908 or later), for simplicity, Windows 10 April 2018 Update (1803) will be checked unless product version check would be better (I think I might as well use product version route).

Thanks.