Support for suggestions in Windows 10 apps

[josephsl]

Hi,

I ran into this issue while writing an app module for Windows 10's MSN Weather app (Microsoft.msn.weather.exe):

If you define a handler for value change (from UIA's value event) as part of an app module or an overlay class (such as search box in Weather app), the first char entered isn't handled correctly (it might be my end).

STR:

1. Run an app such as Weather app.
2. Define an overlay class for search box and define a handler for value change event.
3. With search box empty, enter some characters.

Expected: NVDA should run our custom handling routine for value change, such as locating controller for ID.
Actual: first char isn't handled correctly.

Test code path: in valueChange handler, I asked NVDA to print how many UI elements are affected by search term entry (this is what controller for ID is for). Execution ran thus:

1. Search field is empty.
2. First char comes in.

• Expected: Controller for count is printed.
• Actual: This path isn't executed.
  1. Second char comes in.
• Expected: Controller for routine is called a second time.
• Actual: Controller for count is called as though a single char was entered.
  1. This continues whenever new chars are entered.

So it seems valueChange is looking at the second to last char, not the end. This is an edit field with no auto-detection. The complete dev info is:

name: u'Search'
role: ROLE_EDITABLETEXT
states: STATE_FOCUSABLE, STATE_FOCUSED
isFocusable: True
hasFocus: 1
Python object: <NVDAObjects.Dynamic_SearchBoxEditableTextWithoutAutoSelectDetectionUIA object at 0x04CF9170>
Python class mro: (<class 'NVDAObjects.Dynamic_SearchBoxEditableTextWithoutAutoSelectDetectionUIA'>, <class 'appModules.microsoft_msn_weather.SearchBox'>, <class 'NVDAObjects.behaviors.EditableTextWithoutAutoSelectDetection'>, <class 'NVDAObjects.UIA.UIA'>, <class 'NVDAObjects.window.Window'>, <class 'editableText.EditableTextWithoutAutoSelectDetection'>, <class 'NVDAObjects.behaviors.EditableText'>, <class 'editableText.EditableText'>, <class 'NVDAObjects.NVDAObject'>, <class 'baseObject.ScriptableObject'>, <class 'baseObject.AutoPropertyObject'>, <type 'object'>)
description: u''
location: (794, 36, 250, 40)
value: u''
appModule: <'microsoft_msn_weather' (appName u'microsoft_msn_weather', process ID 2472) at address 4c7c2b0>
appModule.productName: u'Weather'
appModule.productVersion: u'4.11.156.0'
TextInfo: <class 'NVDAObjects.UIA.UIATextInfo'>
windowHandle: 1640546
windowClassName: u'Windows.UI.Core.CoreWindow'
windowControlID: 0
windowStyle: 1409286144
windowThreadID: 5876
windowText: u'Weather'
displayText: u''
UIAElement: <POINTER(IUIAutomationElement) ptr=0x625f3a0 at 4f673a0>
UIA automationID: TextBox
UIA frameworkID: XAML
UIA runtimeID: (42, 1640546, 2, 32)
UIA providerDescription: [pid:2472,hwnd:0x0 Main(parent link):Unidentified Provider (unmanaged:Windows.UI.Xaml.dll)]
UIA className: TextBox

SearchBox is there because it's an overlay class.

Thanks.

[jcsteh]

While this does seem odd (and I'd say it's a bug), valueChange is not really semantically correct here. You probably want textChange, but we don't currently listen for that. Have you tested this with AccEvent? If you do see a valueChange there for the first typed character, we should perhaps look at this, but if you don't, there's nothing for us to do.

That said, if the reason for this is search suggestions, you shouldn't need to watch for either of these events anyway. Are you certain that the selected suggestion item doesn't fire an ElementSelected event? They're supposed to use the same pattern used in the Start Menu; see the code in appModules.searchui.SuggestionListItem. Note that you may need to restart NVDA due to the weird UIA event loss issues.

[josephsl]

Hi,

Accevent does say Value Value event, and suggestions list (not the one found in Start menu) does not raise selection event. I did see text change event as well.
Thanks.

[jcsteh]

After some discussion with @josephsl, I'm morphing this issue to cover suggestions in Windows 10 apps in general rather than dealing with the valueChange event, since I don't think the valueChange event is the best way to do this.

We already support suggestions in the Start Menu search box. (Currently, this code is in the searchui app module.) However, suggestions are supported elsewhere in Windows 10; e.g. the search box in the Settings app. Microsoft have unified the accessibility support for these suggestions. We should support this everywhere, probably by implementing support in the base UIA object.

Notes about the implementation:

1. When suggestions appear or disappear, a controllerFor property change event is fired on the search box.
   • In the Start Menu, when suggestions appear, the first suggestion is selected. However, this isn't true for other suggestions; e.g. the search box in the Settings app. Therefore, we should use this controllerForChange event to detect the appearance of suggestions and report appropriately so the user knows they have appeared.
2. When a suggestion is selected, an elementSelected event is fired on it.
   • We should handle this as we do in searchui.
3. When a search yields no results (e.g. type "asdf" in the Settings search box), a message appears informing that there are no results. At this point, there is no controllerFor on the search box. However, a liveRegionChange event is fired on the message.
   • We should handle liveRegionChange events, which will also help other cases. However, we don't want to handle these for Edge and IE because live region events are broken there (and for IE, we have MSHTML based live region support). We can probably just override the event with a no-op for those cases.

[josephsl]

Hi,

Design specs:

1. Create NVDAObjects.UIA.SuggestionsListItem that's literally a copy of searchui.SuggestionsListItem.
2. The million dollar question is which lists are suggestions?

Issues to consider:

• Apart from Start menu, known suggestions list have standardized on SuggestionsList as UIA Automation ID, whereas Start menu only has capital on L. One could mitigate this via str.lower.
• As noted above, some suggestions containers fire selection event automatically, whereas others don't.

PR and use case on its way. Thanks.

[josephsl]

Hi,

I'm thinking live region support should be a separate branch, but do agree that we need to keep an eye on controller for.

For item 2, done as part of the pull request. For item 1:

1. When controller for says there is a UI element that needs attention, NVDA might want to say, "suggestions open" or "suggestions" or something short yet meaningful. On the second thought, perhaps letting NVDA say, "results available" or "results found" might be better.
2. When controller for says the element in question is gone, NVDA might want to say, "suggestions closed" or "results closed".
3. An alternative is sounds (this may involve adding a config value in Object Presentation dialog).

The question becomes where to place this - in the base class or as an event handler for a search box. It may not be advisable to put this in behaviors, as we want to target Win10/UIA for now unless IAccessible has a similar method to controller for property event.

Thanks.

[josephsl]

Hi,

Item 1 (handling Controller For property) done as part of latest commit. I'd like to gather opinions as to how appearance/disappearance of suggestions should be announced (I'm leaning towards providing a combo box in object presentation that'll let users hear this via a message, sounds or turn it off altogether).

Thanks.

[josephsl]

Hi,

Last thing: should users be given a choice as to how they'd like to be informed about appearance of suggestions, to be categorized under Object Presentation dialog? If so, I'm considering using suggestions sounds recorded by @derekriemer if this is the case. Thanks.

[michaelDCurran]

@josephsl: could you please merge the NVDA branch called pr6241WithMiscDeps into this branch, thereby providing the suggestion sounds in miscDeps, and then I'll incubate this pr.

[josephsl]

Done, hopefully I did the right thing this time. Thanks. From: Michael Curran [mailto:notifications@github.com] Sent: Wednesday, May 10, 2017 11:34 PM To: nvaccess/nvda <nvda@noreply.github.com> Cc: Joseph Lee <joseph.lee22590@gmail.com>; Mention <mention@noreply.github.com> Subject: Re: [nvaccess/nvda] Support for suggestions in Windows 10 apps (#6241) @josephsl <https://github.com/josephsl> : could you please merge the NVDA branch called pr6241WithMiscDeps into this branch, thereby providing the suggestion sounds in miscDeps, and then I'll incubate this pr. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#6241 (comment)> , or mute the thread <https://github.com/notifications/unsubscribe-auth/AHgLkD1ftw7_ielwZMfc7Ee_rgEMfqx0ks5r4qu8gaJpZM4JdeVr> .

[josephsl]

Hi,

An anoying issue found: in some cases, Start menu does not pop up and announced. If this happens, an UnboundLocalError exception is thrown in NVDAObjects.UIA.init line 758 (suggestions list checker). The solution (which is below) is to move the parentElement checker inside the try-except block where the parent element is fetched.

Thanks.

[josephsl]

Hi,

The search field implementation isn't limited to a single line search fields: one way this is used is via at mentions in Mail app.

STR:

1. From Mail app, compose a message.
   While writing a message, type the at sign. The controller for event is fired.

Expected: NVDA may wish to play the suggestions sound.
Actual: no sound.
Possible solution: we may as well rename UIA.SearchField to UIA.UIAEditableTextWithSuggestions and let SearchField become a subclass of this, but we run into inheritance/MRO pollution. This is assuming that controller for list will be updated when suggestions are opened and closed, but we know at least one control that doesn't follow this (Edge's address omnibar). If it's too late for now, I'd be happy to do a follow-up PR with this change later in 2017.

@michaelDCurran, any thoughts? Thanks.

[bramd]

I only tried the suggestions in the start menu so far, but based on that I think this needs some more work for braille.

There is a message "Suggestions" shown when the list is opened. This is very annoying, because you have to click on that or wait until it disappears to see what the suggestion is. Furthermore, you just see the text field with the first suggestion and you miss additional info like the type of suggestion (app/setting etc). Due to the cursor at the end of the edit field, you might have to scroll to read the first part of a suggestion. Since braille just displays the text field, I guess this also breaks if the first suggestion isn't selected automatically in other cases.

After selecting a suggestion, the "Suggestions closed" message appears, I also think this is unnescesary. Usually my apps start faster than the message timout, so I have to click this away manually.

If we improve braille, there are a few questions:

• Do we show the suggestion, or also the user input as well? If so, how should we present that?
• Do we want to give an explicit message for opening/closing or should that be implicit? We don't have messages for other sounds such as browse/focus mode either.

Probably @LeonarddeR and @dkager have input on this as well?

[josephsl]

Hi, I added those messages in braille to embrace deaf-blind users. In this case, it’d be best to have a discussion with @jcsteh. Thanks. From: Bram Duvigneau [mailto:notifications@github.com] Sent: Wednesday, May 31, 2017 1:07 AM To: nvaccess/nvda <nvda@noreply.github.com> Cc: Joseph Lee <joseph.lee22590@gmail.com>; Mention <mention@noreply.github.com> Subject: Re: [nvaccess/nvda] Support for suggestions in Windows 10 apps (#6241) I only tried the suggestions in the start menu so far, but based on that I think this needs some more work for braille. There is a message "Suggestions" shown when the list is opened. This is very annoying, because you have to click on that or wait until it disappears to see what the suggestion is. Furthermore, you just see the text field with the first suggestion and you miss additional info like the type of suggestion (app/setting etc). Due to the cursor at the end of the edit field, you might have to scroll to read the first part of a suggestion. Since braille just displays the text field, I guess this also breaks if the first suggestion isn't selected automatically in other cases. After selecting a suggestion, the "Suggestions closed" message appears, I also think this is unnescesary. Usually my apps start faster than the message timout, so I have to click this away manually. If we improve braille, there are a few questions: * Do we show the suggestion, or also the user input as well? If so, how should we present that? * Do we want to give an explicit message for opening/closing or should that be implicit? We don't have messages for other sounds such as browse/focus mode either. Probably @LeonarddeR <https://github.com/leonardder> and @dkager <https://github.com/dkager> have input on this as well? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#6241 (comment)> , or mute the thread <https://github.com/notifications/unsubscribe-auth/AHgLkIL7pM4JJcykvdTSlSeEgyeZMy0-ks5r_R-7gaJpZM4JdeVr> .

[jcsteh]

I agree suggestions closed is probably unnecessary. However, the message for open may be important in cases where the suggestions open but the first is not selected automatically as in the Settings app. I think we should display selected suggestions as a timed message. That way, you see the suggestion but can still see the input after the message gets dismissed.

[dkager]

We don't have messages for other sounds such as browse/focus mode either.

We do if you disable the sounds. I think that would be good UX for suggestions too, though that implies a setting to turn these sounds of similar to browse mode. Or maybe the two can be rolled into one. Then you'd have either sounds or messages but not both.
Obviously the actual suggestions should be shown somehow regardless of sound.

[josephsl]

Hi, I think it’d be best to work on a separate ticket for sounds/braille messages combo. But as for “suggestions closed”, I think it can be left out for now. Thanks. From: Davy Kager [mailto:notifications@github.com] Sent: Wednesday, May 31, 2017 2:09 AM To: nvaccess/nvda <nvda@noreply.github.com> Cc: Joseph Lee <joseph.lee22590@gmail.com>; Mention <mention@noreply.github.com> Subject: Re: [nvaccess/nvda] Support for suggestions in Windows 10 apps (#6241) We don't have messages for other sounds such as browse/focus mode either. We do if you disable the sounds. I think that would be good UX for suggestions too, though that implies a setting to turn these sounds of similar to browse mode. Or maybe the two can be rolled into one. Then you'd have either sounds or messages but not both. Obviously the actual suggestions should be shown somehow regardless of sound. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#6241 (comment)> , or mute the thread <https://github.com/notifications/unsubscribe-auth/AHgLkCETOIPhjAEH8pMv-g0dzCVV-SQrks5r_S4qgaJpZM4JdeVr> .

[josephsl]

Hi,

A rough way to implement @dkager's suggestion is part of the latest commit. I put together a way to announce name and pos info for suggestion results in braille, which is then fed to braille.handler.message function so it can be shown as a flash message. Ideally, NVDA objects should have a method to return braille flash messages for controls such as for suggestion results.

@jcsteh and @michaelDCurran, another round of review please.

Thanks.

[michaelDCurran]

Fixed by pr #6274