Skip to content

/ nvda

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

support the new emoji panel in windows 10 #7273

Closed
mohdshara opened this issue Jun 11, 2017 · 28 comments
Closed

support the new emoji panel in windows 10 #7273

mohdshara opened this issue Jun 11, 2017 · 28 comments
Labels
feature
Milestone
2018.3

Comments

@mohdshara
Copy link

@mohdshara mohdshara commented Jun 11, 2017

Steps to reproduce:

In windows 10 windows insider build 16215 Microsoft introduced the Emoji panel.

  1. focus on any text field and then press the windows key + ; or + .
  2. the emoji panel appears.

Expected behavior:

NVDA doesn't speak anything, and moving with the right and left keys moves between Emojies but NVDA doesn't read any of them. if you press enter the emoji would get entered as expected and announced.

Actual behavior:

NVDA doesn't tell us that the panel has opned nor does it read when moving between items. Note that Narrator reads those fine as expected.

System configuration:

NVDA version:
NVDA Version: next-14106,059e7f02

NVDA Installed or portable:
installed

Other information:
none needed

Windows version:
Windows Version 1703 (OS Build 16215.1000)

the other questions in the template don't apply here.
@josephsl
Copy link
Collaborator

@josephsl josephsl commented Jun 11, 2017

@jcsteh
Copy link
Contributor

@jcsteh jcsteh commented Jun 11, 2017

I looked into this briefly a couple of days ago. Note that this can be used even if you don't use the touch keyboard.

The emoji panel does work with Narrator.

The emoji panel is beneath a top level window named "Microsoft Text Input Application". This MSDN article about IMEs in Windows Store apps says the following about accessibility:

Implement the following convention to make your IMEs conform to the accessibility requirements and to work with Narrator. To make candidate lists accessible, your IMEs must follow this convention.

  • The candidate list must have a UIA_AutomationIdPropertyId equal to "IME_Candidate_Window" for lists of conversion candidates or "IME_Prediction_Window" for lists of prediction candidates.
  • When the candidate list appears and disappears, it raises events of type UIA_MenuOpenedEventId and UIA_MenuClosedEventId, respectively
  • When the current selected candidate changes, the candidate list raises a UIA_SelectionItem_ElementSelectedEventId. The selected element should have a property UIA_SelectionItemIsSelectedPropertyId equal to TRUE.
  • The UIA_NamePropertyId for each item in the candidate list must be the name of the candidate. Optionally, you can provide additional information to disambiguate candidates through UIA_HelpTextPropertyId.

However, it seems Microsoft doesn't follow their own guidelines here with regard to automation ids:

  1. There are two lists in that window. The first list has an automation id of TEMPLATE_PART_ExpressionFullViewItemsGrid. The second has an automation id of TEMPLATE_PART_ExpressionFullViewGroupsList.
  2. The first list is the list of emoji. The name of each item in this lit is the Unicode representation of the emoji.
  3. The selected item in the first list does change when you use the arrow keys. That means it probably does raise selection events. I think Narrator probably listens for selection events globally, which explains how that works.
  4. The second is the list of categories.Unfortunately, the category list is all private use (and thus not standardised) Unicode characters.
  5. When you press tab, the selected item in the second list changes (corresponding to the selected category).
  6. Note that Narrator doesn't report the categories in any way.

CC @michaelDCurran.
@josephsl
Copy link
Collaborator

@josephsl josephsl commented Jun 12, 2017

@josephsl
Copy link
Collaborator

@josephsl josephsl commented Jun 12, 2017

Hi,

Technical: these are "floating suggestion items" in that they are not part of the app in question. Rather, they appear as part of a floating window with its own app executable name. Numerous solutions come to my mind:

  • #6241: I can add a new FloatingSuggestionsListItem class that will do what SuggestionsListItem does except for checking for controller for event (this is the solution employed in latest WinTenApps snapshot as of June 12th).
  • A request event for UIA objects should be added for the app module for the new XAML input panel. Among other things, this opens up numerous possibilities such as working with floating windows.

Important note: the implementation @jcsteh described and I wrote a workaround for can change in future Insider builds.

Thanks.
@feerrenrut feerrenrut added the feature label Jun 12, 2017
@josephsl
Copy link
Collaborator

@josephsl josephsl commented Jun 25, 2017

Hi,

Although the suggestions list item is in master, I'll hold off until at least two slow ring builds of Fall Creators Update Insider Preview are released in order to gauge how it'll work in the wild. Thanks.
@mohdshara
Copy link
Author

@mohdshara mohdshara commented Jul 8, 2017

in build 16237 some changes were made to the Emoji panel, in particular this may be beneficial to the implementation of this ticket:

"• Have you ever wondered “What the heck is this emoji supposed to be”? Wonder no more! If you hover your mouse over any of the emoji in the Emoji Panel, you’ll now see a handy showing you the exact Unicode character name. Bonus: we’ve added tooltips to the categories so they’re no longer a mystery."
@PratikP1
Copy link

@PratikP1 PratikP1 commented Jul 8, 2017

@josephsl
Copy link
Collaborator

@josephsl josephsl commented Jul 8, 2017

@PratikP1
Copy link

@PratikP1 PratikP1 commented Jul 8, 2017

@jcsteh
Copy link
Contributor

@jcsteh jcsteh commented Jul 10, 2017

@josephsl, I think this would be good to support in core. If you'd be willing to submit a PR, that'd be great. We should think about the best implementation, though:

@josephsl commented on 12 Jun 2017, 12:05 GMT+10:

  • #6241: I can add a new FloatingSuggestionsListItem class that will do what SuggestionsListItem does except for checking for controller for event (this is the solution employed in latest WinTenApps snapshot as of June 12th).
  1. Any thoughts on the likelihood of other floating suggestions like this?
  2. How do you determine what should get this behaviour? Do you use the automation id or similar or is it much more generic? I'd be concerned about unintentional reporting if it were too generic.
  3. Is "suggestions" the right way to think about this? This is probably more like candidates in an input method.
  • A request event for UIA objects should be added for the app module for the new XAML input panel.

Does this need requestEvent? Is it blocked by shouldAcceptEvent by default?

I'm inclined to go more for an app module specific implementation here. Aside from the open questions above, handling the categories properly will require specific code, I think.
@josephsl
Copy link
Collaborator

@josephsl josephsl commented Jul 10, 2017

Hi,

App module method: I'll report back if I find anything new (using slow ring 16232 on my physical desktop, which should make it easier for me to test this).
Candidate or not: I see why it could be labeled that way. If so, I guess it makes implementation super easy unless @michaelDCurran says otherwise.
Other floating suggestions: not that I know of. I'll ask a contact from MS Shell team about this, as well as asking if the Emoji panel implementation is stable, as the answer to that changes when the PR would be ready; if the panel code is stable, I can clean things up and send a PR as early as Tuesday, if not, I can wait until September.

Thanks.
josephsl added a commit to josephsl/nvda that referenced this issue Jul 11, 2017
@josephsl
Composable Shell touch keyboard: support emoji panel in Windows 10 Fa…
963684a 
…ll Creators Update. re nvaccess#7273.

In Windows 10 Fall Creators Update, people using United States English keyboard can use the keyboard to search for and enter emojis (Windows+period or Windows+semicolon). At the moment the app module has been added, with emoji candidates raising UIA element selected event.
Things to be aware of:
* When the emoji panel first opens, the first emoji isn't announced in some cases.
* When searching for emojis, intermediate search results are not announced.
@josephsl josephsl mentioned this issue Jul 11, 2017
Merged
josephsl added a commit to josephsl/nvda that referenced this issue Oct 18, 2017
@josephsl
Emoji panel: take care of recent changes in Fall Creators Update. re n…
d7b0c4b 
…vaccess#7273.

In Fall Creators Update (build 16299), when emoji categories change, the new category fires name change event.
@PratikP1
Copy link

@PratikP1 PratikP1 commented Jan 23, 2018

I wonder if there is progress in adding emoji panel support to NVDA. @josephsl's Win10Apps add-on provides an excellent start. I believe that Microsoft uses a similar mechanism to support for word suggestions for hardware keyboards, a feature introduced in insider build 17035.
@josephsl
Copy link
Collaborator

@josephsl josephsl commented Jan 23, 2018

@PratikP1
Copy link

@PratikP1 PratikP1 commented Jan 23, 2018

@josephsl
Copy link
Collaborator

@josephsl josephsl commented Jan 23, 2018

@PratikP1
Copy link

@PratikP1 PratikP1 commented Jan 23, 2018

@josephsl
Copy link
Collaborator

@josephsl josephsl commented Jan 23, 2018

Hi,

Emoji panel seems stable, but things can change. I'm about to publish a PR for it, hoping it can be included in 2018.1.

Thanks.
@josephsl
Copy link
Collaborator

@josephsl josephsl commented Jan 23, 2018

Hi,

Actually, there is a PR: #7378. Not reviewed in months, partially due to changes during that time. Now that it is stable, I'd like to request review again. Thanks.
@PratikP1
Copy link

@PratikP1 PratikP1 commented Jan 23, 2018

@josephsl
Copy link
Collaborator

@josephsl josephsl commented Jan 23, 2018

@PratikP1
Copy link

@PratikP1 PratikP1 commented Jan 23, 2018

@josephsl
Copy link
Collaborator

@josephsl josephsl commented Jan 23, 2018

@josephsl
Copy link
Collaborator

@josephsl josephsl commented Jan 23, 2018

Hi,

Testing shows the following:

  1. Window open event is fired.
  2. Item selected event is used.
  3. Window opened event isn't fired in 16299 (Version 1709).

In this case, I'll come up with a snapshot of my add-on that'll resolve item announcement bug for 17000 series. Thanks.
josephsl added a commit to josephsl/nvda that referenced this issue Feb 19, 2018
@josephsl
Emoji panel: handle window open event in post-1709 builds. Re nvacces…
a263a3f 
…s#7273.

In Windows 10 17000 and later (particularly after 17063 and later), when emoji panel first opens, window opened event is fired. This than allows NVDA to catch this and treat it as an item selected event, thereby allowing the first category to be announced.
@josephsl josephsl mentioned this issue Apr 18, 2018
Closed
@josephsl josephsl mentioned this issue May 2, 2018
Closed
josephsl added a commit to josephsl/nvda that referenced this issue May 18, 2018
@josephsl
Emoji panel: support redesigned emoji panel in build 17666 and later.…
ca28087 
… Re nvaccess#7273.

In build 17666 and later, emoji panel was redesigned. Instead of refreshing emoji categories when Tab is pressed, one must now select a category before emoji refreshes. This is especially the case for People category where skin tone must be selected afterwards.
For backwards compatibility, continue to support original panel design in Version 1709 and 1803.
josephsl added a commit to josephsl/nvda that referenced this issue Jun 24, 2018
@josephsl
Modern keyboard: suppress 'Microsoft Candidate uI' message and handle…
1b4882f 
… touch keyboard's name change event. Re nvaccess#7273.

In Version 1803 (April 2018 Update), 'Microsoft Candidate UI' is announced whenever  candidates appear. As this is anoying, suppress this.
Also, reported by a user: under some circumstances, touch keyboard keys fire name change event, which results in NVDA announcing characters. Thus suppress this as well.
josephsl added a commit to josephsl/nvda that referenced this issue Jul 13, 2018
@josephsl
Composable Shell touch keyboard: support emoji panel in Windows 10 Fa…
3be8a9c 
…ll Creators Update. re nvaccess#7273.

In Windows 10 Fall Creators Update, people using United States English keyboard can use the keyboard to search for and enter emojis (Windows+period or Windows+semicolon). At the moment the app module has been added, with emoji candidates raising UIA element selected event.
Things to be aware of:
* When the emoji panel first opens, the first emoji isn't announced in some cases.
* When searching for emojis, intermediate search results are not announced.
josephsl added a commit to josephsl/nvda that referenced this issue Jul 13, 2018
@josephsl
Emoji panel: take care of recent changes in Fall Creators Update. re n…
7365920 
…vaccess#7273.

In Fall Creators Update (build 16299), when emoji categories change, the new category fires name change event.
josephsl added a commit to josephsl/nvda that referenced this issue Jul 13, 2018
@josephsl
Emoji panel: handle window open event in post-1709 builds. Re nvacces…
6949431 
…s#7273.

In Windows 10 17000 and later (particularly after 17063 and later), when emoji panel first opens, window opened event is fired. This than allows NVDA to catch this and treat it as an item selected event, thereby allowing the first category to be announced.
josephsl added a commit to josephsl/nvda that referenced this issue Jul 13, 2018
@josephsl
Emoji panel: support redesigned emoji panel in build 17666 and later.…
1d61694 
… Re nvaccess#7273.

In build 17666 and later, emoji panel was redesigned. Instead of refreshing emoji categories when Tab is pressed, one must now select a category before emoji refreshes. This is especially the case for People category where skin tone must be selected afterwards.
For backwards compatibility, continue to support original panel design in Version 1709 and 1803.
josephsl added a commit to josephsl/nvda that referenced this issue Jul 13, 2018
@josephsl
Modern keyboard: suppress 'Microsoft Candidate uI' message and handle…
be522ba 
… touch keyboard's name change event. Re nvaccess#7273.

In Version 1803 (April 2018 Update), 'Microsoft Candidate UI' is announced whenever  candidates appear. As this is anoying, suppress this.
Also, reported by a user: under some circumstances, touch keyboard keys fire name change event, which results in NVDA announcing characters. Thus suppress this as well.
josephsl added a commit to josephsl/nvda that referenced this issue Jul 13, 2018
@josephsl
Modern keyboard: support various changes introduced in build 17700 se…
eeecece 
…ries. Re nvaccess#7273.

Several changes as a result of deploying build 17700 series:
* Clipboard: NVDA will no longer announce label for clipboard candidates list.
* Emoji panel: NVDA will not respond to verbose name change events fired whenever items are selected.
josephsl added a commit to josephsl/nvda that referenced this issue Jul 18, 2018
@josephsl
What's new/new features: added various modern input entries. Re nvacc…
3e96e11 
…ess#7273
josephsl added a commit to josephsl/nvda that referenced this issue Jul 19, 2018
@josephsl
Modern keyboard/emoji panel: catch a weird case where IAccessible con…
8a1b833 
…tent generic client is the parent of the 'selected' item. Re nvaccess#7273.

Reviewed by Mick Curran (NV Access): sometimes, a traceback that ends with 'AttributeError' on line 37 is shown, caused by the fact that sometimes IAccessible content generic client window becomes the parent of the 'selected' item (tree traversal issue). Thus catch this early by making sure the 'parent' is indeed a UIA control.
@nvaccessAuto nvaccessAuto added this to the 2018.3 milestone Jul 19, 2018
michaelDCurran added a commit that referenced this issue Jul 19, 2018
@josephsl @michaelDCurran
Composable Shell touch keyboard: support emoji panel and other featur…
Loading status checks…
e409638 
…es in Windows 10 Fall Creators Update and later. re #7273 (#7378)

* Composable Shell touch keyboard: support emoji panel in Windows 10 Fall Creators Update. re #7273.

In Windows 10 Fall Creators Update, people using United States English keyboard can use the keyboard to search for and enter emojis (Windows+period or Windows+semicolon). At the moment the app module has been added, with emoji candidates raising UIA element selected event.
Things to be aware of:
* When the emoji panel first opens, the first emoji isn't announced in some cases.
* When searching for emojis, intermediate search results are not announced.

* Emoji panel: take care of recent changes in Fall Creators Update. re #7273.

In Fall Creators Update (build 16299), when emoji categories change, the new category fires name change event.

* Readability and copyright year updates.

* Emoji panel: handle window open event in post-1709 builds. Re #7273.

In Windows 10 17000 and later (particularly after 17063 and later), when emoji panel first opens, window opened event is fired. This than allows NVDA to catch this and treat it as an item selected event, thereby allowing the first category to be announced.

* Emoji panel: support redesigned emoji panel in build 17666 and later. Re #7273.

In build 17666 and later, emoji panel was redesigned. Instead of refreshing emoji categories when Tab is pressed, one must now select a category before emoji refreshes. This is especially the case for People category where skin tone must be selected afterwards.
For backwards compatibility, continue to support original panel design in Version 1709 and 1803.

* Modern keyboard: introduce support for hardware input suggestions and cloud clipboard suggestions. Re #8189.

Modern keyboard is not just used for emoji panel entry: in version 1803, hardware keyboard input suggestions are lited there. In Redstone 5, cloud clipboard entries are shown through this panel. Thus add support for both.

* Modern keyboard: suppress 'Microsoft Candidate uI' message and handle touch keyboard's name change event. Re #7273.

In Version 1803 (April 2018 Update), 'Microsoft Candidate UI' is announced whenever  candidates appear. As this is anoying, suppress this.
Also, reported by a user: under some circumstances, touch keyboard keys fire name change event, which results in NVDA announcing characters. Thus suppress this as well.

* Modern keyboard: support various changes introduced in build 17700 series. Re #7273.

Several changes as a result of deploying build 17700 series:
* Clipboard: NVDA will no longer announce label for clipboard candidates list.
* Emoji panel: NVDA will not respond to verbose name change events fired whenever items are selected.

* What's new/new features: added various modern input entries. Re #7273

* Modern keyboard/emoji panel: catch a weird case where IAccessible content generic client is the parent of the 'selected' item. Re #7273.

Reviewed by Mick Curran (NV Access): sometimes, a traceback that ends with 'AttributeError' on line 37 is shown, caused by the fact that sometimes IAccessible content generic client window becomes the parent of the 'selected' item (tree traversal issue). Thus catch this early by making sure the 'parent' is indeed a UIA control.
@ghost
Copy link

@ghost ghost commented Dec 11, 2018

Why left/right/back space/delete does not announce anything? Both JAWS/Narrator does. Should you F*K something when adding/improving something? I'm crying for my $120 donations for the past 4 releases!
@josephsl
Copy link
Collaborator

@josephsl josephsl commented Dec 11, 2018

Hi,

Okay, let's grab something to drink (perhaps soda or wine) and have a deep talk about what's happening. I'm listening - I won't say anything until you tell me the whole thing. Although I think NV Access people might not take it easily, I'm willing to listen to whatever that's in your mind, including rants, insults, and curses.

So, my friend, please tell me what's really bothering you about NVDA right now. Is it about bugs? Something wrong with emoji panel support or something else? Or, if you don't want to talk about what's really going on in public, that's fine.

Sincerely, from a really concerned NVDA contributor
@Qchristensen
Copy link
Contributor

@Qchristensen Qchristensen commented Dec 11, 2018

Hi @armick,

I'd really like to help you with your issue. It's certainly not our intention to break anything when we add or improve features, and I'd really like to get to the bottom of what is not working for you. Are you saying you are having trouble with the emoji panel not working?

Using a slow insider build and trying the emoji panel just now, I am encountering an issue where if you open the emoji panel, then close it (with or without having inserted an emoji), if you then try to open the panel again, it doesn't appear. Is that what you are encountering? That appears to be a bug with Windows itself - I can reproduce it with or without NVDA running. Alt+tabbing to another window and back seems to fix it, or at least make it possible to open the emoji panel again one more time before it happens again.

Please do write to me directly at quentin@nvaccess.org with more information so I can try and help you.
@josephsl
Copy link
Collaborator

@josephsl josephsl commented Dec 11, 2018

Hi,

Hmmm, @armick, is that the issue you're seeing, or something else? If it is exactly what Quentin is saying, chances are that someone will report this to Microsoft.

Technical: I think Quentin is refering to build 18298 (fast ring). If so, this might be caused by UIA window open event not being fired and/or detected by NVDA (I tested this scenario numerous times to no avail). Usually, re-initializing UIA handler and killing modern keyboard app (WindowsInternal.ComposableShell.Experiences.TextInput.InputApp.exe) via Task Manager should resolve this, but apparently, that workaround isn't working for me in build 18298. This also affects cloud clipboard paste command; strangely, dictation and hardware keyboard input suggestions are working. I consider emoji panel support failure to be a high severity bug for Windows 10 users.

However, if your issue is something other than emoji panel support, then please do let us know what's going on and how to reproduce the problem you're seeing.

Thanks.
@Qchristensen
Copy link
Contributor

@Qchristensen Qchristensen commented Dec 11, 2018

Sorry, I should have clarified, I did mean insider build of Windows, but I am using Slow insider build 18272 (which expires in a couple of days apparently. Ridiculous that you get builds expiring, or even messages about builds expiring when there isn't a new build on that ring available. But I digress :) The issue I'm seeing with the emoji panel definitely seems to occur whether or not NVDA is running, so that seems like it might be a different issue to Joseph's.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
feature
Projects
None yet
Milestone
2018.3
Linked pull requests

Successfully merging a pull request may close this issue.

Composable Shell touch keyboard: support emoji panel and other features in Windows 10 Fall Creators Update and later. re #7273
7 participants
@feerrenrut @jcsteh @mohdshara @josephsl @PratikP1 @nvaccessAuto @Qchristensen