New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

support the new emoji panel in windows 10 #7273

Closed
mohdshara opened this Issue Jun 11, 2017 · 23 comments

Comments

Projects
None yet
6 participants
@mohdshara

mohdshara commented Jun 11, 2017

Steps to reproduce:

In windows 10 windows insider build 16215 Microsoft introduced the Emoji panel.

  1. focus on any text field and then press the windows key + ; or + .
  2. the emoji panel appears.

Expected behavior:

NVDA doesn't speak anything, and moving with the right and left keys moves between Emojies but NVDA doesn't read any of them. if you press enter the emoji would get entered as expected and announced.

Actual behavior:

NVDA doesn't tell us that the panel has opned nor does it read when moving between items. Note that Narrator reads those fine as expected.

System configuration:

NVDA version:
NVDA Version: next-14106,059e7f02

NVDA Installed or portable:
installed

Other information:
none needed

Windows version:
Windows Version 1703 (OS Build 16215.1000)

the other questions in the template don't apply here.

@josephsl

This comment has been minimized.

Show comment
Hide comment
@josephsl

josephsl Jun 11, 2017

Collaborator
Collaborator

josephsl commented Jun 11, 2017

@jcsteh

This comment has been minimized.

Show comment
Hide comment
@jcsteh

jcsteh Jun 11, 2017

Contributor

I looked into this briefly a couple of days ago. Note that this can be used even if you don't use the touch keyboard.

The emoji panel does work with Narrator.

The emoji panel is beneath a top level window named "Microsoft Text Input Application". This MSDN article about IMEs in Windows Store apps says the following about accessibility:

Implement the following convention to make your IMEs conform to the accessibility requirements and to work with Narrator. To make candidate lists accessible, your IMEs must follow this convention.

  • The candidate list must have a UIA_AutomationIdPropertyId equal to "IME_Candidate_Window" for lists of conversion candidates or "IME_Prediction_Window" for lists of prediction candidates.
  • When the candidate list appears and disappears, it raises events of type UIA_MenuOpenedEventId and UIA_MenuClosedEventId, respectively
  • When the current selected candidate changes, the candidate list raises a UIA_SelectionItem_ElementSelectedEventId. The selected element should have a property UIA_SelectionItemIsSelectedPropertyId equal to TRUE.
  • The UIA_NamePropertyId for each item in the candidate list must be the name of the candidate. Optionally, you can provide additional information to disambiguate candidates through UIA_HelpTextPropertyId.

However, it seems Microsoft doesn't follow their own guidelines here with regard to automation ids:

  1. There are two lists in that window. The first list has an automation id of TEMPLATE_PART_ExpressionFullViewItemsGrid. The second has an automation id of TEMPLATE_PART_ExpressionFullViewGroupsList.
  2. The first list is the list of emoji. The name of each item in this lit is the Unicode representation of the emoji.
  3. The selected item in the first list does change when you use the arrow keys. That means it probably does raise selection events. I think Narrator probably listens for selection events globally, which explains how that works.
  4. The second is the list of categories.Unfortunately, the category list is all private use (and thus not standardised) Unicode characters.
  5. When you press tab, the selected item in the second list changes (corresponding to the selected category).
  6. Note that Narrator doesn't report the categories in any way.

CC @michaelDCurran.

Contributor

jcsteh commented Jun 11, 2017

I looked into this briefly a couple of days ago. Note that this can be used even if you don't use the touch keyboard.

The emoji panel does work with Narrator.

The emoji panel is beneath a top level window named "Microsoft Text Input Application". This MSDN article about IMEs in Windows Store apps says the following about accessibility:

Implement the following convention to make your IMEs conform to the accessibility requirements and to work with Narrator. To make candidate lists accessible, your IMEs must follow this convention.

  • The candidate list must have a UIA_AutomationIdPropertyId equal to "IME_Candidate_Window" for lists of conversion candidates or "IME_Prediction_Window" for lists of prediction candidates.
  • When the candidate list appears and disappears, it raises events of type UIA_MenuOpenedEventId and UIA_MenuClosedEventId, respectively
  • When the current selected candidate changes, the candidate list raises a UIA_SelectionItem_ElementSelectedEventId. The selected element should have a property UIA_SelectionItemIsSelectedPropertyId equal to TRUE.
  • The UIA_NamePropertyId for each item in the candidate list must be the name of the candidate. Optionally, you can provide additional information to disambiguate candidates through UIA_HelpTextPropertyId.

However, it seems Microsoft doesn't follow their own guidelines here with regard to automation ids:

  1. There are two lists in that window. The first list has an automation id of TEMPLATE_PART_ExpressionFullViewItemsGrid. The second has an automation id of TEMPLATE_PART_ExpressionFullViewGroupsList.
  2. The first list is the list of emoji. The name of each item in this lit is the Unicode representation of the emoji.
  3. The selected item in the first list does change when you use the arrow keys. That means it probably does raise selection events. I think Narrator probably listens for selection events globally, which explains how that works.
  4. The second is the list of categories.Unfortunately, the category list is all private use (and thus not standardised) Unicode characters.
  5. When you press tab, the selected item in the second list changes (corresponding to the selected category).
  6. Note that Narrator doesn't report the categories in any way.

CC @michaelDCurran.

@josephsl

This comment has been minimized.

Show comment
Hide comment
@josephsl

josephsl Jun 12, 2017

Collaborator
Collaborator

josephsl commented Jun 12, 2017

@josephsl

This comment has been minimized.

Show comment
Hide comment
@josephsl

josephsl Jun 12, 2017

Collaborator

Hi,

Technical: these are "floating suggestion items" in that they are not part of the app in question. Rather, they appear as part of a floating window with its own app executable name. Numerous solutions come to my mind:

  • #6241: I can add a new FloatingSuggestionsListItem class that will do what SuggestionsListItem does except for checking for controller for event (this is the solution employed in latest WinTenApps snapshot as of June 12th).
  • A request event for UIA objects should be added for the app module for the new XAML input panel. Among other things, this opens up numerous possibilities such as working with floating windows.

Important note: the implementation @jcsteh described and I wrote a workaround for can change in future Insider builds.

Thanks.

Collaborator

josephsl commented Jun 12, 2017

Hi,

Technical: these are "floating suggestion items" in that they are not part of the app in question. Rather, they appear as part of a floating window with its own app executable name. Numerous solutions come to my mind:

  • #6241: I can add a new FloatingSuggestionsListItem class that will do what SuggestionsListItem does except for checking for controller for event (this is the solution employed in latest WinTenApps snapshot as of June 12th).
  • A request event for UIA objects should be added for the app module for the new XAML input panel. Among other things, this opens up numerous possibilities such as working with floating windows.

Important note: the implementation @jcsteh described and I wrote a workaround for can change in future Insider builds.

Thanks.

@feerrenrut feerrenrut added the feature label Jun 12, 2017

@josephsl

This comment has been minimized.

Show comment
Hide comment
@josephsl

josephsl Jun 25, 2017

Collaborator

Hi,

Although the suggestions list item is in master, I'll hold off until at least two slow ring builds of Fall Creators Update Insider Preview are released in order to gauge how it'll work in the wild. Thanks.

Collaborator

josephsl commented Jun 25, 2017

Hi,

Although the suggestions list item is in master, I'll hold off until at least two slow ring builds of Fall Creators Update Insider Preview are released in order to gauge how it'll work in the wild. Thanks.

@mohdshara

This comment has been minimized.

Show comment
Hide comment
@mohdshara

mohdshara Jul 8, 2017

in build 16237 some changes were made to the Emoji panel, in particular this may be beneficial to the implementation of this ticket:

"• Have you ever wondered “What the heck is this emoji supposed to be”? Wonder no more! If you hover your mouse over any of the emoji in the Emoji Panel, you’ll now see a handy showing you the exact Unicode character name. Bonus: we’ve added tooltips to the categories so they’re no longer a mystery."

mohdshara commented Jul 8, 2017

in build 16237 some changes were made to the Emoji panel, in particular this may be beneficial to the implementation of this ticket:

"• Have you ever wondered “What the heck is this emoji supposed to be”? Wonder no more! If you hover your mouse over any of the emoji in the Emoji Panel, you’ll now see a handy showing you the exact Unicode character name. Bonus: we’ve added tooltips to the categories so they’re no longer a mystery."

@PratikP1

This comment has been minimized.

Show comment
Hide comment
@PratikP1

PratikP1 Jul 8, 2017

PratikP1 commented Jul 8, 2017

@josephsl

This comment has been minimized.

Show comment
Hide comment
@josephsl

josephsl Jul 8, 2017

Collaborator
Collaborator

josephsl commented Jul 8, 2017

@PratikP1

This comment has been minimized.

Show comment
Hide comment
@PratikP1

PratikP1 Jul 8, 2017

PratikP1 commented Jul 8, 2017

@jcsteh

This comment has been minimized.

Show comment
Hide comment
@jcsteh

jcsteh Jul 10, 2017

Contributor

@josephsl, I think this would be good to support in core. If you'd be willing to submit a PR, that'd be great. We should think about the best implementation, though:

@josephsl commented on 12 Jun 2017, 12:05 GMT+10:

  • #6241: I can add a new FloatingSuggestionsListItem class that will do what SuggestionsListItem does except for checking for controller for event (this is the solution employed in latest WinTenApps snapshot as of June 12th).
  1. Any thoughts on the likelihood of other floating suggestions like this?
  2. How do you determine what should get this behaviour? Do you use the automation id or similar or is it much more generic? I'd be concerned about unintentional reporting if it were too generic.
  3. Is "suggestions" the right way to think about this? This is probably more like candidates in an input method.
  • A request event for UIA objects should be added for the app module for the new XAML input panel.

Does this need requestEvent? Is it blocked by shouldAcceptEvent by default?

I'm inclined to go more for an app module specific implementation here. Aside from the open questions above, handling the categories properly will require specific code, I think.

Contributor

jcsteh commented Jul 10, 2017

@josephsl, I think this would be good to support in core. If you'd be willing to submit a PR, that'd be great. We should think about the best implementation, though:

@josephsl commented on 12 Jun 2017, 12:05 GMT+10:

  • #6241: I can add a new FloatingSuggestionsListItem class that will do what SuggestionsListItem does except for checking for controller for event (this is the solution employed in latest WinTenApps snapshot as of June 12th).
  1. Any thoughts on the likelihood of other floating suggestions like this?
  2. How do you determine what should get this behaviour? Do you use the automation id or similar or is it much more generic? I'd be concerned about unintentional reporting if it were too generic.
  3. Is "suggestions" the right way to think about this? This is probably more like candidates in an input method.
  • A request event for UIA objects should be added for the app module for the new XAML input panel.

Does this need requestEvent? Is it blocked by shouldAcceptEvent by default?

I'm inclined to go more for an app module specific implementation here. Aside from the open questions above, handling the categories properly will require specific code, I think.

@josephsl

This comment has been minimized.

Show comment
Hide comment
@josephsl

josephsl Jul 10, 2017

Collaborator

Hi,

App module method: I'll report back if I find anything new (using slow ring 16232 on my physical desktop, which should make it easier for me to test this).
Candidate or not: I see why it could be labeled that way. If so, I guess it makes implementation super easy unless @michaelDCurran says otherwise.
Other floating suggestions: not that I know of. I'll ask a contact from MS Shell team about this, as well as asking if the Emoji panel implementation is stable, as the answer to that changes when the PR would be ready; if the panel code is stable, I can clean things up and send a PR as early as Tuesday, if not, I can wait until September.

Thanks.

Collaborator

josephsl commented Jul 10, 2017

Hi,

App module method: I'll report back if I find anything new (using slow ring 16232 on my physical desktop, which should make it easier for me to test this).
Candidate or not: I see why it could be labeled that way. If so, I guess it makes implementation super easy unless @michaelDCurran says otherwise.
Other floating suggestions: not that I know of. I'll ask a contact from MS Shell team about this, as well as asking if the Emoji panel implementation is stable, as the answer to that changes when the PR would be ready; if the panel code is stable, I can clean things up and send a PR as early as Tuesday, if not, I can wait until September.

Thanks.

josephsl added a commit to josephsl/nvda that referenced this issue Jul 11, 2017

Composable Shell touch keyboard: support emoji panel in Windows 10 Fa…
…ll Creators Update. re nvaccess#7273.

In Windows 10 Fall Creators Update, people using United States English keyboard can use the keyboard to search for and enter emojis (Windows+period or Windows+semicolon). At the moment the app module has been added, with emoji candidates raising UIA element selected event.
Things to be aware of:
* When the emoji panel first opens, the first emoji isn't announced in some cases.
* When searching for emojis, intermediate search results are not announced.

josephsl added a commit to josephsl/nvda that referenced this issue Oct 18, 2017

Emoji panel: take care of recent changes in Fall Creators Update. re n…
…vaccess#7273.

In Fall Creators Update (build 16299), when emoji categories change, the new category fires name change event.
@PratikP1

This comment has been minimized.

Show comment
Hide comment
@PratikP1

PratikP1 Jan 23, 2018

I wonder if there is progress in adding emoji panel support to NVDA. @josephsl's Win10Apps add-on provides an excellent start. I believe that Microsoft uses a similar mechanism to support for word suggestions for hardware keyboards, a feature introduced in insider build 17035.

PratikP1 commented Jan 23, 2018

I wonder if there is progress in adding emoji panel support to NVDA. @josephsl's Win10Apps add-on provides an excellent start. I believe that Microsoft uses a similar mechanism to support for word suggestions for hardware keyboards, a feature introduced in insider build 17035.

@josephsl

This comment has been minimized.

Show comment
Hide comment
@josephsl

josephsl Jan 23, 2018

Collaborator
Collaborator

josephsl commented Jan 23, 2018

@PratikP1

This comment has been minimized.

Show comment
Hide comment
@PratikP1

PratikP1 Jan 23, 2018

PratikP1 commented Jan 23, 2018

@josephsl

This comment has been minimized.

Show comment
Hide comment
@josephsl

josephsl Jan 23, 2018

Collaborator
Collaborator

josephsl commented Jan 23, 2018

@PratikP1

This comment has been minimized.

Show comment
Hide comment
@PratikP1

PratikP1 Jan 23, 2018

PratikP1 commented Jan 23, 2018

@josephsl

This comment has been minimized.

Show comment
Hide comment
@josephsl

josephsl Jan 23, 2018

Collaborator

Hi,

Emoji panel seems stable, but things can change. I'm about to publish a PR for it, hoping it can be included in 2018.1.

Thanks.

Collaborator

josephsl commented Jan 23, 2018

Hi,

Emoji panel seems stable, but things can change. I'm about to publish a PR for it, hoping it can be included in 2018.1.

Thanks.

@josephsl

This comment has been minimized.

Show comment
Hide comment
@josephsl

josephsl Jan 23, 2018

Collaborator

Hi,

Actually, there is a PR: #7378. Not reviewed in months, partially due to changes during that time. Now that it is stable, I'd like to request review again. Thanks.

Collaborator

josephsl commented Jan 23, 2018

Hi,

Actually, there is a PR: #7378. Not reviewed in months, partially due to changes during that time. Now that it is stable, I'd like to request review again. Thanks.

@PratikP1

This comment has been minimized.

Show comment
Hide comment
@PratikP1

PratikP1 Jan 23, 2018

PratikP1 commented Jan 23, 2018

@josephsl

This comment has been minimized.

Show comment
Hide comment
@josephsl

josephsl Jan 23, 2018

Collaborator
Collaborator

josephsl commented Jan 23, 2018

@PratikP1

This comment has been minimized.

Show comment
Hide comment
@PratikP1

PratikP1 Jan 23, 2018

PratikP1 commented Jan 23, 2018

@josephsl

This comment has been minimized.

Show comment
Hide comment
@josephsl

josephsl Jan 23, 2018

Collaborator
Collaborator

josephsl commented Jan 23, 2018

@josephsl

This comment has been minimized.

Show comment
Hide comment
@josephsl

josephsl Jan 23, 2018

Collaborator

Hi,

Testing shows the following:

  1. Window open event is fired.
  2. Item selected event is used.
  3. Window opened event isn't fired in 16299 (Version 1709).

In this case, I'll come up with a snapshot of my add-on that'll resolve item announcement bug for 17000 series. Thanks.

Collaborator

josephsl commented Jan 23, 2018

Hi,

Testing shows the following:

  1. Window open event is fired.
  2. Item selected event is used.
  3. Window opened event isn't fired in 16299 (Version 1709).

In this case, I'll come up with a snapshot of my add-on that'll resolve item announcement bug for 17000 series. Thanks.

josephsl added a commit to josephsl/nvda that referenced this issue Feb 19, 2018

Emoji panel: handle window open event in post-1709 builds. Re nvacces…
…s#7273.

In Windows 10 17000 and later (particularly after 17063 and later), when emoji panel first opens, window opened event is fired. This than allows NVDA to catch this and treat it as an item selected event, thereby allowing the first category to be announced.

josephsl added a commit to josephsl/nvda that referenced this issue May 18, 2018

Emoji panel: support redesigned emoji panel in build 17666 and later.…
… Re nvaccess#7273.

In build 17666 and later, emoji panel was redesigned. Instead of refreshing emoji categories when Tab is pressed, one must now select a category before emoji refreshes. This is especially the case for People category where skin tone must be selected afterwards.
For backwards compatibility, continue to support original panel design in Version 1709 and 1803.

josephsl added a commit to josephsl/nvda that referenced this issue Jun 24, 2018

Modern keyboard: suppress 'Microsoft Candidate uI' message and handle…
… touch keyboard's name change event. Re nvaccess#7273.

In Version 1803 (April 2018 Update), 'Microsoft Candidate UI' is announced whenever  candidates appear. As this is anoying, suppress this.
Also, reported by a user: under some circumstances, touch keyboard keys fire name change event, which results in NVDA announcing characters. Thus suppress this as well.

josephsl added a commit to josephsl/nvda that referenced this issue Jul 13, 2018

Composable Shell touch keyboard: support emoji panel in Windows 10 Fa…
…ll Creators Update. re nvaccess#7273.

In Windows 10 Fall Creators Update, people using United States English keyboard can use the keyboard to search for and enter emojis (Windows+period or Windows+semicolon). At the moment the app module has been added, with emoji candidates raising UIA element selected event.
Things to be aware of:
* When the emoji panel first opens, the first emoji isn't announced in some cases.
* When searching for emojis, intermediate search results are not announced.

josephsl added a commit to josephsl/nvda that referenced this issue Jul 13, 2018

Emoji panel: take care of recent changes in Fall Creators Update. re n…
…vaccess#7273.

In Fall Creators Update (build 16299), when emoji categories change, the new category fires name change event.

josephsl added a commit to josephsl/nvda that referenced this issue Jul 13, 2018

Emoji panel: handle window open event in post-1709 builds. Re nvacces…
…s#7273.

In Windows 10 17000 and later (particularly after 17063 and later), when emoji panel first opens, window opened event is fired. This than allows NVDA to catch this and treat it as an item selected event, thereby allowing the first category to be announced.

josephsl added a commit to josephsl/nvda that referenced this issue Jul 13, 2018

Emoji panel: support redesigned emoji panel in build 17666 and later.…
… Re nvaccess#7273.

In build 17666 and later, emoji panel was redesigned. Instead of refreshing emoji categories when Tab is pressed, one must now select a category before emoji refreshes. This is especially the case for People category where skin tone must be selected afterwards.
For backwards compatibility, continue to support original panel design in Version 1709 and 1803.

josephsl added a commit to josephsl/nvda that referenced this issue Jul 13, 2018

Modern keyboard: suppress 'Microsoft Candidate uI' message and handle…
… touch keyboard's name change event. Re nvaccess#7273.

In Version 1803 (April 2018 Update), 'Microsoft Candidate UI' is announced whenever  candidates appear. As this is anoying, suppress this.
Also, reported by a user: under some circumstances, touch keyboard keys fire name change event, which results in NVDA announcing characters. Thus suppress this as well.

josephsl added a commit to josephsl/nvda that referenced this issue Jul 13, 2018

Modern keyboard: support various changes introduced in build 17700 se…
…ries. Re nvaccess#7273.

Several changes as a result of deploying build 17700 series:
* Clipboard: NVDA will no longer announce label for clipboard candidates list.
* Emoji panel: NVDA will not respond to verbose name change events fired whenever items are selected.

josephsl added a commit to josephsl/nvda that referenced this issue Jul 19, 2018

Modern keyboard/emoji panel: catch a weird case where IAccessible con…
…tent generic client is the parent of the 'selected' item. Re nvaccess#7273.

Reviewed by Mick Curran (NV Access): sometimes, a traceback that ends with 'AttributeError' on line 37 is shown, caused by the fact that sometimes IAccessible content generic client window becomes the parent of the 'selected' item (tree traversal issue). Thus catch this early by making sure the 'parent' is indeed a UIA control.

@nvaccessAuto nvaccessAuto added this to the 2018.3 milestone Jul 19, 2018

michaelDCurran added a commit that referenced this issue Jul 19, 2018

Composable Shell touch keyboard: support emoji panel and other featur…
…es in Windows 10 Fall Creators Update and later. re #7273 (#7378)

* Composable Shell touch keyboard: support emoji panel in Windows 10 Fall Creators Update. re #7273.

In Windows 10 Fall Creators Update, people using United States English keyboard can use the keyboard to search for and enter emojis (Windows+period or Windows+semicolon). At the moment the app module has been added, with emoji candidates raising UIA element selected event.
Things to be aware of:
* When the emoji panel first opens, the first emoji isn't announced in some cases.
* When searching for emojis, intermediate search results are not announced.

* Emoji panel: take care of recent changes in Fall Creators Update. re #7273.

In Fall Creators Update (build 16299), when emoji categories change, the new category fires name change event.

* Readability and copyright year updates.

* Emoji panel: handle window open event in post-1709 builds. Re #7273.

In Windows 10 17000 and later (particularly after 17063 and later), when emoji panel first opens, window opened event is fired. This than allows NVDA to catch this and treat it as an item selected event, thereby allowing the first category to be announced.

* Emoji panel: support redesigned emoji panel in build 17666 and later. Re #7273.

In build 17666 and later, emoji panel was redesigned. Instead of refreshing emoji categories when Tab is pressed, one must now select a category before emoji refreshes. This is especially the case for People category where skin tone must be selected afterwards.
For backwards compatibility, continue to support original panel design in Version 1709 and 1803.

* Modern keyboard: introduce support for hardware input suggestions and cloud clipboard suggestions. Re #8189.

Modern keyboard is not just used for emoji panel entry: in version 1803, hardware keyboard input suggestions are lited there. In Redstone 5, cloud clipboard entries are shown through this panel. Thus add support for both.

* Modern keyboard: suppress 'Microsoft Candidate uI' message and handle touch keyboard's name change event. Re #7273.

In Version 1803 (April 2018 Update), 'Microsoft Candidate UI' is announced whenever  candidates appear. As this is anoying, suppress this.
Also, reported by a user: under some circumstances, touch keyboard keys fire name change event, which results in NVDA announcing characters. Thus suppress this as well.

* Modern keyboard: support various changes introduced in build 17700 series. Re #7273.

Several changes as a result of deploying build 17700 series:
* Clipboard: NVDA will no longer announce label for clipboard candidates list.
* Emoji panel: NVDA will not respond to verbose name change events fired whenever items are selected.

* What's new/new features: added various modern input entries. Re #7273

* Modern keyboard/emoji panel: catch a weird case where IAccessible content generic client is the parent of the 'selected' item. Re #7273.

Reviewed by Mick Curran (NV Access): sometimes, a traceback that ends with 'AttributeError' on line 37 is shown, caused by the fact that sometimes IAccessible content generic client window becomes the parent of the 'selected' item (tree traversal issue). Thus catch this early by making sure the 'parent' is indeed a UIA control.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment