New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support for suggestions in Windows 10 apps #6241

Closed
josephsl opened this Issue Aug 5, 2016 · 18 comments

Comments

Projects
None yet
5 participants
@josephsl
Collaborator

josephsl commented Aug 5, 2016

Hi,

I ran into this issue while writing an app module for Windows 10's MSN Weather app (Microsoft.msn.weather.exe):

If you define a handler for value change (from UIA's value event) as part of an app module or an overlay class (such as search box in Weather app), the first char entered isn't handled correctly (it might be my end).

STR:

  1. Run an app such as Weather app.
  2. Define an overlay class for search box and define a handler for value change event.
  3. With search box empty, enter some characters.

Expected: NVDA should run our custom handling routine for value change, such as locating controller for ID.
Actual: first char isn't handled correctly.

Test code path: in valueChange handler, I asked NVDA to print how many UI elements are affected by search term entry (this is what controller for ID is for). Execution ran thus:

  1. Search field is empty.
  2. First char comes in.
  • Expected: Controller for count is printed.
  • Actual: This path isn't executed.
    1. Second char comes in.
  • Expected: Controller for routine is called a second time.
  • Actual: Controller for count is called as though a single char was entered.
    1. This continues whenever new chars are entered.

So it seems valueChange is looking at the second to last char, not the end. This is an edit field with no auto-detection. The complete dev info is:

name: u'Search'
role: ROLE_EDITABLETEXT
states: STATE_FOCUSABLE, STATE_FOCUSED
isFocusable: True
hasFocus: 1
Python object: <NVDAObjects.Dynamic_SearchBoxEditableTextWithoutAutoSelectDetectionUIA object at 0x04CF9170>
Python class mro: (<class 'NVDAObjects.Dynamic_SearchBoxEditableTextWithoutAutoSelectDetectionUIA'>, <class 'appModules.microsoft_msn_weather.SearchBox'>, <class 'NVDAObjects.behaviors.EditableTextWithoutAutoSelectDetection'>, <class 'NVDAObjects.UIA.UIA'>, <class 'NVDAObjects.window.Window'>, <class 'editableText.EditableTextWithoutAutoSelectDetection'>, <class 'NVDAObjects.behaviors.EditableText'>, <class 'editableText.EditableText'>, <class 'NVDAObjects.NVDAObject'>, <class 'baseObject.ScriptableObject'>, <class 'baseObject.AutoPropertyObject'>, <type 'object'>)
description: u''
location: (794, 36, 250, 40)
value: u''
appModule: <'microsoft_msn_weather' (appName u'microsoft_msn_weather', process ID 2472) at address 4c7c2b0>
appModule.productName: u'Weather'
appModule.productVersion: u'4.11.156.0'
TextInfo: <class 'NVDAObjects.UIA.UIATextInfo'>
windowHandle: 1640546
windowClassName: u'Windows.UI.Core.CoreWindow'
windowControlID: 0
windowStyle: 1409286144
windowThreadID: 5876
windowText: u'Weather'
displayText: u''
UIAElement: <POINTER(IUIAutomationElement) ptr=0x625f3a0 at 4f673a0>
UIA automationID: TextBox
UIA frameworkID: XAML
UIA runtimeID: (42, 1640546, 2, 32)
UIA providerDescription: [pid:2472,hwnd:0x0 Main(parent link):Unidentified Provider (unmanaged:Windows.UI.Xaml.dll)]
UIA className: TextBox

SearchBox is there because it's an overlay class.

Thanks.

@jcsteh

This comment has been minimized.

Show comment
Hide comment
@jcsteh

jcsteh Aug 8, 2016

Contributor

While this does seem odd (and I'd say it's a bug), valueChange is not really semantically correct here. You probably want textChange, but we don't currently listen for that. Have you tested this with AccEvent? If you do see a valueChange there for the first typed character, we should perhaps look at this, but if you don't, there's nothing for us to do.

That said, if the reason for this is search suggestions, you shouldn't need to watch for either of these events anyway. Are you certain that the selected suggestion item doesn't fire an ElementSelected event? They're supposed to use the same pattern used in the Start Menu; see the code in appModules.searchui.SuggestionListItem. Note that you may need to restart NVDA due to the weird UIA event loss issues.

Contributor

jcsteh commented Aug 8, 2016

While this does seem odd (and I'd say it's a bug), valueChange is not really semantically correct here. You probably want textChange, but we don't currently listen for that. Have you tested this with AccEvent? If you do see a valueChange there for the first typed character, we should perhaps look at this, but if you don't, there's nothing for us to do.

That said, if the reason for this is search suggestions, you shouldn't need to watch for either of these events anyway. Are you certain that the selected suggestion item doesn't fire an ElementSelected event? They're supposed to use the same pattern used in the Start Menu; see the code in appModules.searchui.SuggestionListItem. Note that you may need to restart NVDA due to the weird UIA event loss issues.

@josephsl

This comment has been minimized.

Show comment
Hide comment
@josephsl

josephsl Aug 8, 2016

Collaborator

Hi,

Accevent does say Value Value event, and suggestions list (not the one found in Start menu) does not raise selection event. I did see text change event as well.
Thanks.

Collaborator

josephsl commented Aug 8, 2016

Hi,

Accevent does say Value Value event, and suggestions list (not the one found in Start menu) does not raise selection event. I did see text change event as well.
Thanks.

@jcsteh jcsteh changed the title from UIA/value change event: the first character typed into an empty edit field does not result in this event being handled correctly to Support for suggestions in Windows 10 apps Aug 8, 2016

@jcsteh

This comment has been minimized.

Show comment
Hide comment
@jcsteh

jcsteh Aug 8, 2016

Contributor

After some discussion with @josephsl, I'm morphing this issue to cover suggestions in Windows 10 apps in general rather than dealing with the valueChange event, since I don't think the valueChange event is the best way to do this.

We already support suggestions in the Start Menu search box. (Currently, this code is in the searchui app module.) However, suggestions are supported elsewhere in Windows 10; e.g. the search box in the Settings app. Microsoft have unified the accessibility support for these suggestions. We should support this everywhere, probably by implementing support in the base UIA object.

Notes about the implementation:

  1. When suggestions appear or disappear, a controllerFor property change event is fired on the search box.
    • In the Start Menu, when suggestions appear, the first suggestion is selected. However, this isn't true for other suggestions; e.g. the search box in the Settings app. Therefore, we should use this controllerForChange event to detect the appearance of suggestions and report appropriately so the user knows they have appeared.
  2. When a suggestion is selected, an elementSelected event is fired on it.
    • We should handle this as we do in searchui.
  3. When a search yields no results (e.g. type "asdf" in the Settings search box), a message appears informing that there are no results. At this point, there is no controllerFor on the search box. However, a liveRegionChange event is fired on the message.
    • We should handle liveRegionChange events, which will also help other cases. However, we don't want to handle these for Edge and IE because live region events are broken there (and for IE, we have MSHTML based live region support). We can probably just override the event with a no-op for those cases.
Contributor

jcsteh commented Aug 8, 2016

After some discussion with @josephsl, I'm morphing this issue to cover suggestions in Windows 10 apps in general rather than dealing with the valueChange event, since I don't think the valueChange event is the best way to do this.

We already support suggestions in the Start Menu search box. (Currently, this code is in the searchui app module.) However, suggestions are supported elsewhere in Windows 10; e.g. the search box in the Settings app. Microsoft have unified the accessibility support for these suggestions. We should support this everywhere, probably by implementing support in the base UIA object.

Notes about the implementation:

  1. When suggestions appear or disappear, a controllerFor property change event is fired on the search box.
    • In the Start Menu, when suggestions appear, the first suggestion is selected. However, this isn't true for other suggestions; e.g. the search box in the Settings app. Therefore, we should use this controllerForChange event to detect the appearance of suggestions and report appropriately so the user knows they have appeared.
  2. When a suggestion is selected, an elementSelected event is fired on it.
    • We should handle this as we do in searchui.
  3. When a search yields no results (e.g. type "asdf" in the Settings search box), a message appears informing that there are no results. At this point, there is no controllerFor on the search box. However, a liveRegionChange event is fired on the message.
    • We should handle liveRegionChange events, which will also help other cases. However, we don't want to handle these for Edge and IE because live region events are broken there (and for IE, we have MSHTML based live region support). We can probably just override the event with a no-op for those cases.
@josephsl

This comment has been minimized.

Show comment
Hide comment
@josephsl

josephsl Aug 17, 2016

Collaborator

Hi,

Design specs:

  1. Create NVDAObjects.UIA.SuggestionsListItem that's literally a copy of searchui.SuggestionsListItem.
  2. The million dollar question is which lists are suggestions?

Issues to consider:

  • Apart from Start menu, known suggestions list have standardized on SuggestionsList as UIA Automation ID, whereas Start menu only has capital on L. One could mitigate this via str.lower.
  • As noted above, some suggestions containers fire selection event automatically, whereas others don't.

PR and use case on its way. Thanks.

Collaborator

josephsl commented Aug 17, 2016

Hi,

Design specs:

  1. Create NVDAObjects.UIA.SuggestionsListItem that's literally a copy of searchui.SuggestionsListItem.
  2. The million dollar question is which lists are suggestions?

Issues to consider:

  • Apart from Start menu, known suggestions list have standardized on SuggestionsList as UIA Automation ID, whereas Start menu only has capital on L. One could mitigate this via str.lower.
  • As noted above, some suggestions containers fire selection event automatically, whereas others don't.

PR and use case on its way. Thanks.

@josephsl josephsl self-assigned this Aug 17, 2016

josephsl added a commit to josephsl/nvda that referenced this issue Aug 17, 2016

UIA objects: introduce a global suggestion list item class for Window…
…s 10. re #6241

Windows 10 uses suggestions list for vairous things, including Start menu suggestions, Store recommendations, Settings app and others. Thus introduce NVDAObjects.UIA.SuggestionListItem, which derives its power from searchui.py version (searchui.py version is gone).
@josephsl

This comment has been minimized.

Show comment
Hide comment
@josephsl

josephsl Aug 17, 2016

Collaborator

Hi,

I'm thinking live region support should be a separate branch, but do agree that we need to keep an eye on controller for.

For item 2, done as part of the pull request. For item 1:

  1. When controller for says there is a UI element that needs attention, NVDA might want to say, "suggestions open" or "suggestions" or something short yet meaningful. On the second thought, perhaps letting NVDA say, "results available" or "results found" might be better.
  2. When controller for says the element in question is gone, NVDA might want to say, "suggestions closed" or "results closed".
  3. An alternative is sounds (this may involve adding a config value in Object Presentation dialog).

The question becomes where to place this - in the base class or as an event handler for a search box. It may not be advisable to put this in behaviors, as we want to target Win10/UIA for now unless IAccessible has a similar method to controller for property event.

Thanks.

Collaborator

josephsl commented Aug 17, 2016

Hi,

I'm thinking live region support should be a separate branch, but do agree that we need to keep an eye on controller for.

For item 2, done as part of the pull request. For item 1:

  1. When controller for says there is a UI element that needs attention, NVDA might want to say, "suggestions open" or "suggestions" or something short yet meaningful. On the second thought, perhaps letting NVDA say, "results available" or "results found" might be better.
  2. When controller for says the element in question is gone, NVDA might want to say, "suggestions closed" or "results closed".
  3. An alternative is sounds (this may involve adding a config value in Object Presentation dialog).

The question becomes where to place this - in the base class or as an event handler for a search box. It may not be advisable to put this in behaviors, as we want to target Win10/UIA for now unless IAccessible has a similar method to controller for property event.

Thanks.

josephsl added a commit to josephsl/nvda that referenced this issue Aug 18, 2016

UIA: add support for Controller For property. re #6241
Controller For property is used by an element to let another element depend on values given by the source element, useful if suggestions should be listed for a search field among other possibilities.
For now, Controller For property has been added to property to event names map in UIA Handler. A UIA object that handles this event (a search box) is next.

josephsl added a commit to josephsl/nvda that referenced this issue Aug 18, 2016

josephsl added a commit to josephsl/nvda that referenced this issue Aug 18, 2016

UIA: add support for Controller For property. re #6241
Controller For property is used by an element to let another element depend on values given by the source element, useful if suggestions should be listed for a search field among other possibilities.
For now, Controller For property has been added to property to event names map in UIA Handler. A UIA object that handles this event (a search box) is next.

josephsl added a commit to josephsl/nvda that referenced this issue Aug 18, 2016

NVDAObjects.UIA: introduce Search Field that'll handle controller for…
… and announce appearance of suggestions. re #6241

Coming from Windows 10 App Essentials add-on: a Search Field is now available that'll detect controller for property event and announce either 'suggestions' or 'suggestions closed' if suggestions appear or disappear, respectively. This takes care of both Start Menu and other fields in Windows 10 (works best in Anniversary Update and later).
@josephsl

This comment has been minimized.

Show comment
Hide comment
@josephsl

josephsl Aug 18, 2016

Collaborator

Hi,

Item 1 (handling Controller For property) done as part of latest commit. I'd like to gather opinions as to how appearance/disappearance of suggestions should be announced (I'm leaning towards providing a combo box in object presentation that'll let users hear this via a message, sounds or turn it off altogether).

Thanks.

Collaborator

josephsl commented Aug 18, 2016

Hi,

Item 1 (handling Controller For property) done as part of latest commit. I'd like to gather opinions as to how appearance/disappearance of suggestions should be announced (I'm leaning towards providing a combo box in object presentation that'll let users hear this via a message, sounds or turn it off altogether).

Thanks.

josephsl added a commit to josephsl/nvda that referenced this issue Oct 27, 2016

Search Field: check if search field is focused before asking for cont…
…roller for object. re #6241

Review from Mick Curran (NV Access): Do not waste a function call for fetching focused element. Besides, make sure to check if the focused control is the search field before proceeding to announce appearance of suggestions.

josephsl added a commit to josephsl/nvda that referenced this issue Jan 18, 2017

UIA objects: introduce a global suggestion list item class for Window…
…s 10. re #6241

Windows 10 uses suggestions list for vairous things, including Start menu suggestions, Store recommendations, Settings app and others. Thus introduce NVDAObjects.UIA.SuggestionListItem, which derives its power from searchui.py version (searchui.py version is gone).

josephsl added a commit to josephsl/nvda that referenced this issue Jan 18, 2017

UIA: add support for Controller For property. re #6241
Controller For property is used by an element to let another element depend on values given by the source element, useful if suggestions should be listed for a search field among other possibilities.
For now, Controller For property has been added to property to event names map in UIA Handler. A UIA object that handles this event (a search box) is next.

josephsl added a commit to josephsl/nvda that referenced this issue Jan 18, 2017

josephsl added a commit to josephsl/nvda that referenced this issue Jan 18, 2017

UIA: add support for Controller For property. re #6241
Controller For property is used by an element to let another element depend on values given by the source element, useful if suggestions should be listed for a search field among other possibilities.
For now, Controller For property has been added to property to event names map in UIA Handler. A UIA object that handles this event (a search box) is next.

josephsl added a commit to josephsl/nvda that referenced this issue Jan 18, 2017

NVDAObjects.UIA: introduce Search Field that'll handle controller for…
… and announce appearance of suggestions. re #6241

Coming from Windows 10 App Essentials add-on: a Search Field is now available that'll detect controller for property event and announce either 'suggestions' or 'suggestions closed' if suggestions appear or disappear, respectively. This takes care of both Start Menu and other fields in Windows 10 (works best in Anniversary Update and later).

josephsl added a commit to josephsl/nvda that referenced this issue Jan 18, 2017

Search Field: check if search field is focused before asking for cont…
…roller for object. re #6241

Review from Mick Curran (NV Access): Do not waste a function call for fetching focused element. Besides, make sure to check if the focused control is the search field before proceeding to announce appearance of suggestions.

josephsl added a commit to josephsl/nvda that referenced this issue Jan 18, 2017

Suggestion: a new behavior to allow NVDA to detect and announce sugge…
…stions. re #6241.

Reviewed by Mick Curran (NV Access): announcement of search suggestion is something that NVDA should handle for other API's, such as suggestions in Firefox address bar, search suggestions in universal apps and so on.
A new behavior named 'Suggestion' has been added that allows subclasses to provide custom routines when suggestions appear and disappear. This is handled by event_suggestionOpened and event_suggestionClosed, and by default NVDA will speak and braille this event.

josephsl added a commit to josephsl/nvda that referenced this issue Jan 18, 2017

UIA objects/Search field: use the new NVDAObjects.behaviors.Suggestio…
…n to announce suggestion appearance. re #6241.

Reviewed by Mick Curran (NV Access): Based on the new Suggestion behavior mix-in, it is now possible for various objects to provide custom routines to let users know the appearance of suggestions. Thus UIA/Search Field is the first object to use this routine.

josephsl added a commit to josephsl/nvda that referenced this issue Jan 18, 2017

UIA objects/suggestions list: use raw UIA base tree walker to obtain …
…suggestion list (parent). re #6241.

Reviewed by Mick Curran (NV Access): it is better to use raw UIA for obtaining parent element. However, one must be careful to catch COM and value errors (COM because the element might not be there, and value because NULL pointer access is logged). The raw UIA method was also recommended by Derek Riemer.
@josephsl

This comment has been minimized.

Show comment
Hide comment
@josephsl

josephsl Jan 18, 2017

Collaborator

Hi,

Last thing: should users be given a choice as to how they'd like to be informed about appearance of suggestions, to be categorized under Object Presentation dialog? If so, I'm considering using suggestions sounds recorded by @derekriemer if this is the case. Thanks.

Collaborator

josephsl commented Jan 18, 2017

Hi,

Last thing: should users be given a choice as to how they'd like to be informed about appearance of suggestions, to be categorized under Object Presentation dialog? If so, I'm considering using suggestions sounds recorded by @derekriemer if this is the case. Thanks.

josephsl added a commit to josephsl/nvda that referenced this issue Jan 18, 2017

UIA objects/Search field: don't allow suggestions detection be announ…
…ced in Windows 10 Start menu for consistency with earlier versions of Windows. re #6241.

iN Windows 10 Start menu, suggestions are announced automaticlaly, so no need to provide suggestion announcements.

josephsl added a commit to josephsl/nvda that referenced this issue Jan 18, 2017

NVDAObjects/behaviors: rename 'Suggestion' to 'EditableTextWithSugges…
…tions' so the behavior can be better described. re #6241.

If one inherits from 'Suggestion', the overall impression would be that the field is only going to display suggestions nad nothing else. Many suggestions are shown when text is entered, thus it is better to say 'EditableTextWithSuggestions' to better reflect what the behavior actually does.
Also, a sound will be played to let users know the appearance of suggestions. (default vlaue).

josephsl added a commit to josephsl/nvda that referenced this issue Jan 19, 2017

Auto-suggestions: add a combo box in object presentation settings to …
…configure how auto-suggestions should be announced. re #6241

josephsl added a commit to josephsl/nvda that referenced this issue Jan 19, 2017

User guide: document auto-suggestions notification setting. re #6241.
In the user guide, an explanatory text has been added to describe what auto-suggestions are.

josephsl added a commit to josephsl/nvda that referenced this issue Jan 19, 2017

Auto-suggestions: add a combo box in object presentation settings to …
…configure how auto-suggestions should be announced. re #6241

josephsl added a commit to josephsl/nvda that referenced this issue Jan 19, 2017

User guide: document auto-suggestions notification setting. re #6241.
In the user guide, an explanatory text has been added to describe what auto-suggestions are.

josephsl added a commit to josephsl/nvda that referenced this issue Jan 19, 2017

Auto-suggestions: either play a sound or do nothing. re #6241.
Comments from Mick Curran and Jamie Teh (NV Access): no need for a separate message option when announcing appearance of auto-suggestions, as the sound cue will let the user of this fact. However, braille users should be notified of this regardless of this flag being turned on (deaf-blind users should be notified of this).
User guide and settings dialog: changed the control type and label for auto-suggestions setting to reflect change in behavior.

josephsl added a commit to josephsl/nvda that referenced this issue Jan 19, 2017

Search suggestions: allow Windows 10 Start menu and Edge to provide s…
…earch suggestions. re #6241.

Reviewed by Mick Curran (NV Access): allow Start menu search box and Edge's address omnibar to participate in providing auto-suggestions announcement.
@michaelDCurran

This comment has been minimized.

Show comment
Hide comment
@michaelDCurran

michaelDCurran May 11, 2017

Contributor

@josephsl: could you please merge the NVDA branch called pr6241WithMiscDeps into this branch, thereby providing the suggestion sounds in miscDeps, and then I'll incubate this pr.

Contributor

michaelDCurran commented May 11, 2017

@josephsl: could you please merge the NVDA branch called pr6241WithMiscDeps into this branch, thereby providing the suggestion sounds in miscDeps, and then I'll incubate this pr.

@josephsl

This comment has been minimized.

Show comment
Hide comment
@josephsl

josephsl May 11, 2017

Collaborator
Collaborator

josephsl commented May 11, 2017

@josephsl

This comment has been minimized.

Show comment
Hide comment
@josephsl

josephsl May 11, 2017

Collaborator

Hi,

An anoying issue found: in some cases, Start menu does not pop up and announced. If this happens, an UnboundLocalError exception is thrown in NVDAObjects.UIA.init line 758 (suggestions list checker). The solution (which is below) is to move the parentElement checker inside the try-except block where the parent element is fetched.

Thanks.

Collaborator

josephsl commented May 11, 2017

Hi,

An anoying issue found: in some cases, Start menu does not pop up and announced. If this happens, an UnboundLocalError exception is thrown in NVDAObjects.UIA.init line 758 (suggestions list checker). The solution (which is below) is to move the parentElement checker inside the try-except block where the parent element is fetched.

Thanks.

josephsl added a commit to josephsl/nvda that referenced this issue May 11, 2017

NVDAObjects.UIA: catch UnboundLocalError for parentElement if parentE…
…lement fetcher fails. re #6241.

In some cases, when Start menu opens, it isn't announced by NVDA. As a result, parent element fetcher fails when trying to instantiate suggestions list item, with a traceback that ends with UnboundLocalError. Catch this by moving the SuggestionsListItem selector to inside of the try block.
@josephsl

This comment has been minimized.

Show comment
Hide comment
@josephsl

josephsl May 14, 2017

Collaborator

Hi,

The search field implementation isn't limited to a single line search fields: one way this is used is via at mentions in Mail app.

STR:

  1. From Mail app, compose a message.
    While writing a message, type the at sign. The controller for event is fired.

Expected: NVDA may wish to play the suggestions sound.
Actual: no sound.
Possible solution: we may as well rename UIA.SearchField to UIA.UIAEditableTextWithSuggestions and let SearchField become a subclass of this, but we run into inheritance/MRO pollution. This is assuming that controller for list will be updated when suggestions are opened and closed, but we know at least one control that doesn't follow this (Edge's address omnibar). If it's too late for now, I'd be happy to do a follow-up PR with this change later in 2017.

@michaelDCurran, any thoughts? Thanks.

Collaborator

josephsl commented May 14, 2017

Hi,

The search field implementation isn't limited to a single line search fields: one way this is used is via at mentions in Mail app.

STR:

  1. From Mail app, compose a message.
    While writing a message, type the at sign. The controller for event is fired.

Expected: NVDA may wish to play the suggestions sound.
Actual: no sound.
Possible solution: we may as well rename UIA.SearchField to UIA.UIAEditableTextWithSuggestions and let SearchField become a subclass of this, but we run into inheritance/MRO pollution. This is assuming that controller for list will be updated when suggestions are opened and closed, but we know at least one control that doesn't follow this (Edge's address omnibar). If it's too late for now, I'd be happy to do a follow-up PR with this change later in 2017.

@michaelDCurran, any thoughts? Thanks.

@bramd

This comment has been minimized.

Show comment
Hide comment
@bramd

bramd May 31, 2017

Contributor

I only tried the suggestions in the start menu so far, but based on that I think this needs some more work for braille.

There is a message "Suggestions" shown when the list is opened. This is very annoying, because you have to click on that or wait until it disappears to see what the suggestion is. Furthermore, you just see the text field with the first suggestion and you miss additional info like the type of suggestion (app/setting etc). Due to the cursor at the end of the edit field, you might have to scroll to read the first part of a suggestion. Since braille just displays the text field, I guess this also breaks if the first suggestion isn't selected automatically in other cases.

After selecting a suggestion, the "Suggestions closed" message appears, I also think this is unnescesary. Usually my apps start faster than the message timout, so I have to click this away manually.

If we improve braille, there are a few questions:

  • Do we show the suggestion, or also the user input as well? If so, how should we present that?
  • Do we want to give an explicit message for opening/closing or should that be implicit? We don't have messages for other sounds such as browse/focus mode either.

Probably @leonardder and @dkager have input on this as well?

Contributor

bramd commented May 31, 2017

I only tried the suggestions in the start menu so far, but based on that I think this needs some more work for braille.

There is a message "Suggestions" shown when the list is opened. This is very annoying, because you have to click on that or wait until it disappears to see what the suggestion is. Furthermore, you just see the text field with the first suggestion and you miss additional info like the type of suggestion (app/setting etc). Due to the cursor at the end of the edit field, you might have to scroll to read the first part of a suggestion. Since braille just displays the text field, I guess this also breaks if the first suggestion isn't selected automatically in other cases.

After selecting a suggestion, the "Suggestions closed" message appears, I also think this is unnescesary. Usually my apps start faster than the message timout, so I have to click this away manually.

If we improve braille, there are a few questions:

  • Do we show the suggestion, or also the user input as well? If so, how should we present that?
  • Do we want to give an explicit message for opening/closing or should that be implicit? We don't have messages for other sounds such as browse/focus mode either.

Probably @leonardder and @dkager have input on this as well?

@josephsl

This comment has been minimized.

Show comment
Hide comment
@josephsl

josephsl May 31, 2017

Collaborator
Collaborator

josephsl commented May 31, 2017

@jcsteh

This comment has been minimized.

Show comment
Hide comment
@jcsteh

jcsteh May 31, 2017

Contributor
Contributor

jcsteh commented May 31, 2017

@dkager

This comment has been minimized.

Show comment
Hide comment
@dkager

dkager May 31, 2017

Collaborator

We don't have messages for other sounds such as browse/focus mode either.

We do if you disable the sounds. I think that would be good UX for suggestions too, though that implies a setting to turn these sounds of similar to browse mode. Or maybe the two can be rolled into one. Then you'd have either sounds or messages but not both.
Obviously the actual suggestions should be shown somehow regardless of sound.

Collaborator

dkager commented May 31, 2017

We don't have messages for other sounds such as browse/focus mode either.

We do if you disable the sounds. I think that would be good UX for suggestions too, though that implies a setting to turn these sounds of similar to browse mode. Or maybe the two can be rolled into one. Then you'd have either sounds or messages but not both.
Obviously the actual suggestions should be shown somehow regardless of sound.

@josephsl

This comment has been minimized.

Show comment
Hide comment
@josephsl

josephsl May 31, 2017

Collaborator
Collaborator

josephsl commented May 31, 2017

josephsl added a commit to josephsl/nvda that referenced this issue May 31, 2017

EditableTextWithSuggestions: remove braille message shown when sugges…
…tions close for consistency with other situations (such as browse mode toggle). re #6241.

josephsl added a commit to josephsl/nvda that referenced this issue May 31, 2017

NVDAObjects.UIA.SuggestionListItem: braille suggestion results as fla…
…sh messages. re #6241.

Suggestion from Davy Kager: provide a way to let braille users read search suggestion items. This is done by emulating some parts of speech.SpeakObjectProperties except the name and position info map will be fetched (position info map fetching is contingent on whether report position info setting is enabled from Object Presentation dialog). Ideally, NVDA objects should have a way to fetch braille flash messages for controls.
Also reworded docstring for SuggestionListItem so it cna include other UIA-based suggestion list items such as Windows 8.x search results.
@josephsl

This comment has been minimized.

Show comment
Hide comment
@josephsl

josephsl May 31, 2017

Collaborator

Hi,

A rough way to implement @dkager's suggestion is part of the latest commit. I put together a way to announce name and pos info for suggestion results in braille, which is then fed to braille.handler.message function so it can be shown as a flash message. Ideally, NVDA objects should have a method to return braille flash messages for controls such as for suggestion results.

@jcsteh and @michaelDCurran, another round of review please.

Thanks.

Collaborator

josephsl commented May 31, 2017

Hi,

A rough way to implement @dkager's suggestion is part of the latest commit. I put together a way to announce name and pos info for suggestion results in braille, which is then fed to braille.handler.message function so it can be shown as a flash message. Ideally, NVDA objects should have a method to return braille flash messages for controls such as for suggestion results.

@jcsteh and @michaelDCurran, another round of review please.

Thanks.

josephsl added a commit to josephsl/nvda that referenced this issue May 31, 2017

NVDAObjects.UIA.SuggestionsListItem: flash suggestion results in brai…
…lle. re #6241, #6414.

Instead of constructing the likely flash message, use a function used as part of Core issue 6414, which is much simpler than constructing the flash message from scratch.

josephsl added a commit to josephsl/nvda that referenced this issue Jun 15, 2017

UIA/SuggestionListItem/searchui: also classify Start search results c…
…ontext menu as a suggestions list item. re #6241.

Oddly, the same behavior that's applied to suggestion items must work in Start suggestion's context menu, otherwise menu items will not be announced.

michaelDCurran added a commit that referenced this issue Jun 23, 2017

UIA objects: introduce a global suggestion list item class for Window…
…s 10. re #6241 (#6274)

* UIA handler: add proper copyright header (based on Git log archive)

* UIA objects: introduce a global suggestion list item class for Windows 10. re #6241

Windows 10 uses suggestions list for vairious things, including Start menu suggestions, Store recommendations, Settings app and others. Thus introduce NVDAObjects.UIA.SuggestionListItem, which derives its power from searchui.py version (searchui.py version is gone).

* SearchUI: remove unneeded imports

* UIA: add support for Controller For property. re #6241

Controller For property is used by an element to let another element depend on values given by the source element, useful if suggestions should be listed for a search field among other possibilities.
For now, Controller For property has been added to property to event names map in UIA Handler. A UIA object that handles this event (a search box) is next.

* Revert "UIA: add support for Controller For property. re #6241"

This reverts commit a3b1ce5.

* UIA: add support for Controller For property. re #6241

Controller For property is used by an element to let another element depend on values given by the source element, useful if suggestions should be listed for a search field among other possibilities.
For now, Controller For property has been added to property to event names map in UIA Handler. A UIA object that handles this event (a search box) is next.

* NVDAObjects.UIA: introduce Search Field that'll handle controller for and announce appearance of suggestions. re #6241

Coming from Windows 10 App Essentials add-on: a Search Field is now available that'll detect controller for property event and announce either 'suggestions' or 'suggestions closed' if suggestions appear or disappear, respectively. This takes care of both Start Menu and other fields in Windows 10 (works best in Anniversary Update and later).

* Search Field: check if search field is focused before asking for controller for object. re #6241

Review from Mick Curran (NV Access): Do not waste a function call for fetching focused element. Besides, make sure to check if the focused control is the search field before proceeding to announce appearance of suggestions.

* Update copyright years, add a docstring to UIA objects (__init__).

* Suggestion: a new behavior to allow NVDA to detect and announce suggestions. re #6241.

Reviewed by Mick Curran (NV Access): announcement of search suggestion is something that NVDA should handle for other API's, such as suggestions in Firefox address bar, search suggestions in universal apps and so on.
A new behavior named 'Suggestion' has been added that allows subclasses to provide custom routines when suggestions appear and disappear. This is handled by event_suggestionOpened and event_suggestionClosed, and by default NVDA will speak and braille this event.

* UIA objects/Search field: use the new NVDAObjects.behaviors.Suggestion to announce suggestion appearance. re #6241.

Reviewed by Mick Curran (NV Access): Based on the new Suggestion behavior mix-in, it is now possible for various objects to provide custom routines to let users know the appearance of suggestions. Thus UIA/Search Field is the first object to use this routine.

* UIA objects/suggestions list: use raw UIA base tree walker to obtain suggestion list (parent). re #6241.

Reviewed by Mick Curran (NV Access): it is better to use raw UIA for obtaining parent element. However, one must be careful to catch COM and value errors (COM because the element might not be there, and value because NULL pointer access is logged). The raw UIA method was also recommended by Derek Riemer.

* UIA objects: updated comments, removed reference to Redstone.

* UIA objects/Search field: don't allow suggestions detection be announced in Windows 10 Start menu for consistency with earlier versions of Windows. re #6241.

iN Windows 10 Start menu, suggestions are announced automaticlaly, so no need to provide suggestion announcements.

* NVDAObjects/behaviors: rename 'Suggestion' to 'EditableTextWithSuggestions' so the behavior can be better described. re #6241.

If one inherits from 'Suggestion', the overall impression would be that the field is only going to display suggestions nad nothing else. Many suggestions are shown when text is entered, thus it is better to say 'EditableTextWithSuggestions' to better reflect what the behavior actually does.
Also, a sound will be played to let users know the appearance of suggestions. (default vlaue).

* Auto-suggestions: add a combo box in object presentation settings to configure how auto-suggestions should be announced. re #6241

* User guide: document auto-suggestions notification setting. re #6241.

In the user guide, an explanatory text has been added to describe what auto-suggestions are.

* Auto-suggestions: either play a sound or do nothing. re #6241.

Comments from Mick Curran and Jamie Teh (NV Access): no need for a separate message option when announcing appearance of auto-suggestions, as the sound cue will let the user of this fact. However, braille users should be notified of this regardless of this flag being turned on (deaf-blind users should be notified of this).
User guide and settings dialog: changed the control type and label for auto-suggestions setting to reflect change in behavior.

* Search suggestions: allow Windows 10 Start menu and Edge to provide search suggestions. re #6241.

Reviewed by Mick Curran (NV Access): allow Start menu search box and Edge's address omnibar to participate in providing auto-suggestions announcement.

* Config: auto-suggestion setting is now part of configspec module. re #6241

* Update to miscDeps containing auto suggestions sounds.

* NVDAObjects.UIA: catch UnboundLocalError for parentElement if parentElement fetcher fails. re #6241.

In some cases, when Start menu opens, it isn't announced by NVDA. As a result, parent element fetcher fails when trying to instantiate suggestions list item, with a traceback that ends with UnboundLocalError. Catch this by moving the SuggestionsListItem selector to inside of the try block.

* EditableTextWithSuggestions: remove braille message shown when suggestions close for consistency with other situations (such as browse mode toggle). re #6241.

* NVDAObjects.UIA.SuggestionListItem: braille suggestion results as flash messages. re #6241.

Suggestion from Davy Kager: provide a way to let braille users read search suggestion items. This is done by emulating some parts of speech.SpeakObjectProperties except the name and position info map will be fetched (position info map fetching is contingent on whether report position info setting is enabled from Object Presentation dialog). Ideally, NVDA objects should have a way to fetch braille flash messages for controls.
Also reworded docstring for SuggestionListItem so it cna include other UIA-based suggestion list items such as Windows 8.x search results.

* NVDAObjects.UIA.SuggestionsListItem: flash suggestion results in braille. re #6241, #6414.

Instead of constructing the likely flash message, use a function used as part of Core issue 6414, which is much simpler than constructing the flash message from scratch.

* UIA/SuggestionListItem/searchui: also classify Start search results context menu as a suggestions list item. re #6241.

Oddly, the same behavior that's applied to suggestion items must work in Start suggestion's context menu, otherwise menu items will not be announced.
@michaelDCurran

This comment has been minimized.

Show comment
Hide comment
@michaelDCurran

michaelDCurran Jun 23, 2017

Contributor

Fixed by pr #6274

Contributor

michaelDCurran commented Jun 23, 2017

Fixed by pr #6274

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment