Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

UIA: add support for layout invalidated event, used to handle auto-suggest accessibility #12790

Closed
josephsl opened this issue Aug 28, 2021 · 1 comment · Fixed by #12791
Closed
Milestone

Comments

@josephsl
Copy link
Collaborator

josephsl commented Aug 28, 2021

Hi,

Different issues talk about this, therefore this issue will serve as a summary issue for a possible pull request:

Overview:

Apps can provide search fields where users can search for terms. The best known example is Start menu/screen where people can search a variety of things, including files, web searches, ans system settings.

Over the years, NVDA added support for search field implementations. These include auto-complete announcements in web browsers, Start menu search results announcement, and even python Console input suggestions.

Is your feature request related to a problem? Please describe.

Back in 2017, NvDA added ability to play a sound when suggestions appear. This is done by listneing to UIA controller for event where the event returns an array of UIA elements that the focused element relies on. For example, in Windows 10/11 Start menu, controller for array for the search box is the results list.

However search suggestions announcement didn't cover a whole class of search fields and their suggestions list friends: those using UIA layout invalidated event. Layout invalidated event (introduced in Windows 8) is used to communicate changes to container elements. The best known application is detecting suggestion count, and one real-life example is Settings app (Windows+I) where a suggestions list uses this event to communicate suggestion count.

In short, the current implementation when encountering this class of search fields/suggestions list combo is:

  1. User types search terms.
  2. Controller for event is fired.
  3. NVDA plays suggestion sound if enabled.
  4. Layout invalidated event is fired.
  5. User types more terms.
  6. Layout invalidated event is fired each time suggestions list changes.
  7. NVDA does nothing.

Describe the solution you'd like

Let NVDA announce at least suggestion count, which is applicable to Windows 8 and later provided that NVDA can be told to detect suggestion lists. Specifically:

  1. Search fields: announce suggestion count #7330: let NVDA announce suggestion count.
  2. Win11: NVDA is not announcing the available suggestions for “Find a setting” search field in settings page  #12758: detect suggestion lists.

Implementation strategy: a real-life implementation is part of Windows App Essentials add-on. Specifically:

  1. Add UIA layout invalidated event to a dictionary of UIA events.
  2. Define an overlay class for search suggestion lists. So far these have the Automation Id of "SuggestionsList".
  3. The overlay class will listen to layout invalidated event, which will simply announce list item count each time it changes.

Describe alternatives you've considered

Leave the code as is. Alternatively, refine controller for event so suggestion count will be announced only when suggestions appear.

Additional context

Implementing auto-suggest: https://docs.microsoft.com/en-us/windows/apps/design/accessibility/accessible-text-requirements#implementing_auto-suggest

Thanks.

@josephsl
Copy link
Collaborator Author

Hi,

One additional task might be to add an option to toggle suggestion count announcement, either a new item or tie into dynamic content changes. I'm fine with either way - the upcoming pull request will leave room for this possibility.

Thanks.

josephsl added a commit to josephsl/nvda that referenced this issue Aug 29, 2021
Recent Windows releases include better support for auto-suggest accessibility. So far there are two lists: one that raises item selected event on the first suggestion (Windows 10/11 Start menu, for example), and ones that uses layout invalidated event instead of selecting the first suggestion (Settings suggestion, for example). The first type was implemented years ago, but the second type isn't. Therefore introduce support for these kinds of suggestions lists, starting with adding UIA layout invalidated event constant (20008) to UIA events map.
josephsl added a commit to josephsl/nvda that referenced this issue Aug 29, 2021
… invalidated event. Re nvaccess#12790.

Introduce an overlay class for suggestions list view which does not select the top suggestion but instead fires layout invalidated event. Examples include modern apps such as Windows 10/11 Settings app, Microsoft Store, and Maps app.
Inside the new suggestions list class, layout invalidated event handler will simly announce child (suggestions) count whenever this event is fired.
josephsl added a commit to josephsl/nvda that referenced this issue Aug 29, 2021
…d event. Re nvaccess#12790.

Suggestions list views with layout invalidated event have UI Automation Id of 'SuggestionsList', not to be confused with Edge-based suggestions list (lowercase s). The overlay class chooser code layout resembles search field -> suggestions list -> suggestion list item to help explain the overall flow of how auto-suggest typically works.
josephsl added a commit to josephsl/nvda that referenced this issue Aug 29, 2021
…ist item and new suggestions list. Re nvaccess#12790.

Clarify that suggestion list item class handles top suggestion being selected automatically.
seanbudd added a commit that referenced this issue Sep 27, 2021
… modern apps on Windows 10 and later (#12791)

Fixes #12790 
Fixes #7330 
Fixes #12758 

Summary of the issue:
Some suggestions list views do not select top suggestion automatically. Instead, UIA layout invalidated event is fired whenever list count and/or content changes.

Description of how this pull request fixes the issue:
Introduce a dedicated UIA overlay class for suggestions list views firing layout invalidated event, with the event handler told to announce suggestion count across modern apps on Windows 10 and later. Although layout invalidated event is supported on Windows 8.x, limit this to Windows 10 and later at first. At the same time, perform bonus lint to (finally) fix Flake8 issues in UIA events map and add annotations (at last).

Manual testing:

Prerequisites: Windows App Essentials add-on must be disabled.

1. Open Start menu.
2. Type something. NVDA will announce the top suggestion.
3. Open Settings app (Windows+I).
4. Type something. NVDA will only play suggestion sound without this pull request, it will also announce suggestion count as user types more terms with the PR applied.

Commits:

* UIA handler: add layout invalidated event constant. Re #12790.

Recent Windows releases include better support for auto-suggest accessibility. So far there are two lists: one that raises item selected event on the first suggestion (Windows 10/11 Start menu, for example), and ones that uses layout invalidated event instead of selecting the first suggestion (Settings suggestion, for example). The first type was implemented years ago, but the second type isn't. Therefore introduce support for these kinds of suggestions lists, starting with adding UIA layout invalidated event constant (20008) to UIA events map.

* NVDAObjects.UIA: introduce SuggestionsList class to handle UIA layout invalidated event. Re #12790.

Introduce an overlay class for suggestions list view which does not select the top suggestion but instead fires layout invalidated event. Examples include modern apps such as Windows 10/11 Settings app, Microsoft Store, and Maps app.
Inside the new suggestions list class, layout invalidated event handler will simly announce child (suggestions) count whenever this event is fired.

* NVDAObjects.UIA: detect suggestions list views with layout invalidated event. Re #12790.

Suggestions list views with layout invalidated event have UI Automation Id of 'SuggestionsList', not to be confused with Edge-based suggestions list (lowercase s). The overlay class chooser code layout resembles search field -> suggestions list -> suggestion list item to help explain the overall flow of how auto-suggest typically works.

* NVDAObjects.UIA: clarify the difference between existing suggestion list item and new suggestions list. Re #12790.

Clarify that suggestion list item class handles top suggestion being selected automatically.

* UIA handler and UIA objects: lint bonus.

* UIA events: the map was (finally) tagged with annotations, a space was (finally) added between event Id's and names, finally resolve Flake8 F405 on event Id's.
* NVDAObjects.UIA.SuggestionListItem: spacing.

* update changes

Co-authored-by: buddsean <sean@nvaccess.org>
@nvaccessAuto nvaccessAuto added this to the 2021.3 milestone Sep 27, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
2 participants