support the new emoji panel in windows 10

[mohdshara]

Steps to reproduce:

In windows 10 windows insider build 16215 Microsoft introduced the Emoji panel.

1. focus on any text field and then press the windows key + ; or + .
2. the emoji panel appears.

Expected behavior:

NVDA doesn't speak anything, and moving with the right and left keys moves between Emojies but NVDA doesn't read any of them. if you press enter the emoji would get entered as expected and announced.

Actual behavior:

NVDA doesn't tell us that the panel has opned nor does it read when moving between items. Note that Narrator reads those fine as expected.

System configuration:

NVDA version:
NVDA Version: next-14106,059e7f02

NVDA Installed or portable:
installed

Other information:
none needed

Windows version:
Windows Version 1703 (OS Build 16215.1000)

the other questions in the template don't apply here.

[josephsl]

Hi, partly because we need to find out if the new XAML touch input panel fires useful events and announce changes if any. CC @michaelDCurran From: Mohammed Al Shara [mailto:notifications@github.com] Sent: Sunday, June 11, 2017 3:24 AM To: nvaccess/nvda <nvda@noreply.github.com> Cc: Subscribed <subscribed@noreply.github.com> Subject: [nvaccess/nvda] support the new emoji panel in windows 10 (#7273) Steps to reproduce: In windows 10 windows insider build 16215 Microsoft introduced the Emoji panel. 1. focus on any text field and then press the windows key + ; or + . 2. the emoji panel appears. Expected behavior: NVDA doesn't speak anything, and moving with the right and left keys moves between Emojies but NVDA doesn't read any of them. if you press enter the emoji would get entered as expected and announced. Actual behavior: NVDA doesn't tell us that the panel has opned nor does it read when moving between items. Note that Narrator reads those fine as expected. System configuration: NVDA version: NVDA Version: next-14106,059e7f02 NVDA Installed or portable: installed Other information: none needed Windows version: Windows Version 1703 (OS Build 16215.1000) the other questions in the template don't apply here. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#7273> , or mute the thread <https://github.com/notifications/unsubscribe-auth/AHgLkAuT1WKls9FDnqfppS_coloDfqWqks5sC8A0gaJpZM4N2WjF> .

[jcsteh]

I looked into this briefly a couple of days ago. Note that this can be used even if you don't use the touch keyboard.

The emoji panel does work with Narrator.

The emoji panel is beneath a top level window named "Microsoft Text Input Application". This MSDN article about IMEs in Windows Store apps says the following about accessibility:

Implement the following convention to make your IMEs conform to the accessibility requirements and to work with Narrator. To make candidate lists accessible, your IMEs must follow this convention.

• The candidate list must have a UIA_AutomationIdPropertyId equal to "IME_Candidate_Window" for lists of conversion candidates or "IME_Prediction_Window" for lists of prediction candidates.
• When the candidate list appears and disappears, it raises events of type UIA_MenuOpenedEventId and UIA_MenuClosedEventId, respectively
• When the current selected candidate changes, the candidate list raises a UIA_SelectionItem_ElementSelectedEventId. The selected element should have a property UIA_SelectionItemIsSelectedPropertyId equal to TRUE.
• The UIA_NamePropertyId for each item in the candidate list must be the name of the candidate. Optionally, you can provide additional information to disambiguate candidates through UIA_HelpTextPropertyId.

However, it seems Microsoft doesn't follow their own guidelines here with regard to automation ids:

1. There are two lists in that window. The first list has an automation id of TEMPLATE_PART_ExpressionFullViewItemsGrid. The second has an automation id of TEMPLATE_PART_ExpressionFullViewGroupsList.
2. The first list is the list of emoji. The name of each item in this lit is the Unicode representation of the emoji.
3. The selected item in the first list does change when you use the arrow keys. That means it probably does raise selection events. I think Narrator probably listens for selection events globally, which explains how that works.
4. The second is the list of categories.Unfortunately, the category list is all private use (and thus not standardised) Unicode characters.
5. When you press tab, the selected item in the second list changes (corresponding to the selected category).
6. Note that Narrator doesn't report the categories in any way.

CC @michaelDCurran.

[josephsl]

Hi, yes, element selected event is indeed raised. I’ll come up with a temporary workaround in the WinTenApps add-on. Thanks. From: James Teh [mailto:notifications@github.com] Sent: Sunday, June 11, 2017 3:52 PM To: nvaccess/nvda <nvda@noreply.github.com> Cc: Joseph Lee <joseph.lee22590@gmail.com>; Comment <comment@noreply.github.com> Subject: Re: [nvaccess/nvda] support the new emoji panel in windows 10 (#7273) I looked into this briefly a couple of days ago. Note that this can be used even if you don't use the touch keyboard. The emoji panel does work with Narrator. The emoji panel is beneath a top level window named "Microsoft Text Input Application". This MSDN article about IMEs in Windows Store apps <https://msdn.microsoft.com/en-us/library/windows/apps/hh967425.aspx#accessibility> says the following about accessibility: Implement the following convention to make your IMEs conform to the accessibility requirements and to work with Narrator. To make candidate lists accessible, your IMEs must follow this convention. * The candidate list must have a UIA_AutomationIdPropertyId equal to "IME_Candidate_Window" for lists of conversion candidates or "IME_Prediction_Window" for lists of prediction candidates. * When the candidate list appears and disappears, it raises events of type UIA_MenuOpenedEventId and UIA_MenuClosedEventId, respectively * When the current selected candidate changes, the candidate list raises a UIA_SelectionItem_ElementSelectedEventId. The selected element should have a property UIA_SelectionItemIsSelectedPropertyId equal to TRUE. * The UIA_NamePropertyId for each item in the candidate list must be the name of the candidate. Optionally, you can provide additional information to disambiguate candidates through UIA_HelpTextPropertyId. However, it seems Microsoft doesn't follow their own guidelines here with regard to automation ids: 1. There are two lists in that window. The first list has an automation id of TEMPLATE_PART_ExpressionFullViewItemsGrid. The second has an automation id of TEMPLATE_PART_ExpressionFullViewGroupsList. 2. The first list is the list of emoji. The name of each item in this lit is the Unicode representation of the emoji. 3. The selected item in the first list does change when you use the arrow keys. That means it probably does raise selection events. I think Narrator probably listens for selection events globally, which explains how that works. 4. The second is the list of categories.Unfortunately, the category list is all private use (and thus not standardised) Unicode characters. 5. When you press tab, the selected item in the second list changes (corresponding to the selected category). 6. Note that Narrator doesn't report the categories in any way. CC @michaelDCurran <https://github.com/michaeldcurran> . — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#7273 (comment)> , or mute the thread <https://github.com/notifications/unsubscribe-auth/AHgLkBvEEImKFNu715dhZZOgr--uliXBks5sDG-VgaJpZM4N2WjF> . <https://github.com/notifications/beacon/AHgLkI3mjWLjQoneQdU1kZ2IiXUqo84gks5sDG-VgaJpZM4N2WjF.gif>

[josephsl]

Hi,

Technical: these are "floating suggestion items" in that they are not part of the app in question. Rather, they appear as part of a floating window with its own app executable name. Numerous solutions come to my mind:

• Support for suggestions in Windows 10 apps #6241: I can add a new FloatingSuggestionsListItem class that will do what SuggestionsListItem does except for checking for controller for event (this is the solution employed in latest WinTenApps snapshot as of June 12th).
• A request event for UIA objects should be added for the app module for the new XAML input panel. Among other things, this opens up numerous possibilities such as working with floating windows.

Important note: the implementation @jcsteh described and I wrote a workaround for can change in future Insider builds.

Thanks.

[josephsl]

Hi,

Although the suggestions list item is in master, I'll hold off until at least two slow ring builds of Fall Creators Update Insider Preview are released in order to gauge how it'll work in the wild. Thanks.

[mohdshara]

in build 16237 some changes were made to the Emoji panel, in particular this may be beneficial to the implementation of this ticket:

"• Have you ever wondered “What the heck is this emoji supposed to be”? Wonder no more! If you hover your mouse over any of the emoji in the Emoji Panel, you’ll now see a handy showing you the exact Unicode character name. Bonus: we’ve added tooltips to the categories so they’re no longer a mystery."

[PratikP1]

Experientially speaking, in build 16237, there appears to be no difference in how Narrator or NVDA treat the emoji panel than originally reported by @jcsteh. I filed a bug in the Feedback Center last week. See the following. https://aka.ms/fk2cna

[josephsl]

Hi, and a possible modification might be in order, as the panel title is announced when I use arrow keys (at least the implementation in my add-on may require some edits). Thanks. From: PratikP1 [mailto:notifications@github.com] Sent: Saturday, July 8, 2017 3:19 AM To: nvaccess/nvda <nvda@noreply.github.com> Cc: Joseph Lee <joseph.lee22590@gmail.com>; Comment <comment@noreply.github.com> Subject: Re: [nvaccess/nvda] support the new emoji panel in windows 10 (#7273) Experientially speaking, in build 16237, there appears to be no difference in how Narrator or NVDA treat the emoji panel than originally reported by @jcsteh. I filed a bug in the Feedback Center last week. See the following. https://aka.ms/fk2cna — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#7273 (comment)> , or mute the thread <https://github.com/notifications/unsubscribe-auth/AHgLkAuAJN3aEW6nDCSwrkhKHjTDMTtHks5sL1ehgaJpZM4N2WjF> .

[PratikP1]

Here is a log snippet of what NVDA says when Win 10 Apps Essential 2017-07-05-dev is running. Window Title is not announced There’s nothing without the add-on running. IO - inputCore.InputManager.executeGesture (12:20:07.668): Input: kb(laptop):windows+. IO - inputCore.InputManager.executeGesture (12:20:09.502): Input: kb(laptop):tab IO - speech.speak (12:20:09.595): Speaking [LangChangeCommand (u'en_US'), u'1 inches'] IO - speech.speak (12:20:09.693): Speaking [LangChangeCommand (u'en_US'), u'blank'] IO - speech.speak (12:20:10.374): Speaking [LangChangeCommand (u'en_US'), u'\U0001f600 1 of 256'] IO - inputCore.InputManager.executeGesture (12:20:12.174): Input: kb(laptop):shift+tab IO - speech.speak (12:20:12.278): Speaking [LangChangeCommand (u'en_US'), u'1 inches'] IO - speech.speak (12:20:12.394): Speaking [LangChangeCommand (u'en_US'), u'blank'] IO - speech.speak (12:20:12.746): Speaking [LangChangeCommand (u'en_US'), u'\U0001f602 1 of 45'] IO - inputCore.InputManager.executeGesture (12:20:14.892): Input: kb(laptop):rightArrow IO - speech._speakSpellingGen (12:20:15.151): Speaking character u'line feed' IO - speech.speak (12:20:15.158): Speaking [LangChangeCommand (u'en_US'), u'\U0001f600 2 of 45'] IO - inputCore.InputManager.executeGesture (12:20:15.964): Input: kb(laptop):rightArrow IO - speech._speakSpellingGen (12:20:16.204): Speaking character u'line feed' IO - speech.speak (12:20:16.213): Speaking [LangChangeCommand (u'en_US'), u'\U0001f601 3 of 45'] IO - inputCore.InputManager.executeGesture (12:20:17.252): Input: kb(laptop):rightArrow IO - speech._speakSpellingGen (12:20:17.484): Speaking character u'line feed' IO - speech.speak (12:20:17.492): Speaking [LangChangeCommand (u'en_US'), u'\U0001f923 4 of 45'] IO - inputCore.InputManager.executeGesture (12:20:17.948): Input: kb(laptop):tab IO - speech.speak (12:20:18.045): Speaking [LangChangeCommand (u'en_US'), u'1 inches'] IO - speech.speak (12:20:18.147): Speaking [LangChangeCommand (u'en_US'), u'blank'] IO - speech.speak (12:20:18.783): Speaking [LangChangeCommand (u'en_US'), u'\U0001f600 1 of 256'] IO - inputCore.InputManager.executeGesture (12:20:20.069): Input: kb(laptop):tab IO - speech.speak (12:20:20.177): Speaking [LangChangeCommand (u'en_US'), u'1 inches'] IO - speech.speak (12:20:20.285): Speaking [LangChangeCommand (u'en_US'), u'blank'] IO - speech.speak (12:20:21.020): Speaking [LangChangeCommand (u'en_US'), u'\U0001f469 1 of 175'] IO - inputCore.InputManager.executeGesture (12:20:22.428): Input: kb(laptop):tab IO - speech.speak (12:20:22.526): Speaking [LangChangeCommand (u'en_US'), u'1 inches'] IO - speech.speak (12:20:22.638): Speaking [LangChangeCommand (u'en_US'), u'blank'] IO - speech.speak (12:20:23.562): Speaking [LangChangeCommand (u'en_US'), u'\U0001f388 1 of 273'] IO - inputCore.InputManager.executeGesture (12:20:25.029): Input: kb(laptop):end IO - speech._speakSpellingGen (12:20:25.796): Speaking character u'line feed' IO - inputCore.InputManager.executeGesture (12:20:27.059): Input: kb(laptop):leftArrow IO - speech._speakSpellingGen (12:20:27.286): Speaking character u'line feed' IO - speech.speak (12:20:27.296): Speaking [LangChangeCommand (u'en_US'), u'\u23f2 272 of 273']

[jcsteh]

@josephsl, I think this would be good to support in core. If you'd be willing to submit a PR, that'd be great. We should think about the best implementation, though:

@josephsl commented on 12 Jun 2017, 12:05 GMT+10:

• Support for suggestions in Windows 10 apps #6241: I can add a new FloatingSuggestionsListItem class that will do what SuggestionsListItem does except for checking for controller for event (this is the solution employed in latest WinTenApps snapshot as of June 12th).

1. Any thoughts on the likelihood of other floating suggestions like this?
2. How do you determine what should get this behaviour? Do you use the automation id or similar or is it much more generic? I'd be concerned about unintentional reporting if it were too generic.
3. Is "suggestions" the right way to think about this? This is probably more like candidates in an input method.

• A request event for UIA objects should be added for the app module for the new XAML input panel.

Does this need requestEvent? Is it blocked by shouldAcceptEvent by default?

I'm inclined to go more for an app module specific implementation here. Aside from the open questions above, handling the categories properly will require specific code, I think.

[josephsl]

Hi,

App module method: I'll report back if I find anything new (using slow ring 16232 on my physical desktop, which should make it easier for me to test this).
Candidate or not: I see why it could be labeled that way. If so, I guess it makes implementation super easy unless @michaelDCurran says otherwise.
Other floating suggestions: not that I know of. I'll ask a contact from MS Shell team about this, as well as asking if the Emoji panel implementation is stable, as the answer to that changes when the PR would be ready; if the panel code is stable, I can clean things up and send a PR as early as Tuesday, if not, I can wait until September.

Thanks.