Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Windows 11/hardware input suggestions and IME: NVDA does not announce top hardware keyboard input suggestions or candidates when IME window opens #16283

Closed
josephsl opened this issue Mar 9, 2024 · 1 comment · Fixed by #16313
Closed

Windows 11/hardware input suggestions and IME: NVDA does not announce top hardware keyboard input suggestions or candidates when IME window opens #16283

josephsl opened this issue Mar 9, 2024 · 1 comment · Fixed by #16313
Labels
feature/i18n Internationalization features p3 https://github.com/nvaccess/nvda/blob/master/projectDocs/issues/triage.md#priority triaged Has been triaged, issue is waiting for implementation.
Milestone
2024.2

Comments

@josephsl
Copy link
Collaborator

josephsl commented Mar 9, 2024

Hi,

Stems from #14509:

Background

With #14509, NVDA will not announce IME entries twice when modern IME opens in Windows 11. Unfortunately, this did not cover the case of hardware keyboard input suggestions where the top result was annoucned in Windows 10 whereas it does not in Windows 11, traced to IME window traversal due to UIA tree differences in modern keyboard/IME window.

Steps to reproduce:

Hardware keyboard input suggestions

  1. Enable hardware keyboard input suggestions in Windows 11 (Settings/Time and language/Typing, enable "Show text suggestions when typing on the physical keyboard").
  2. If using multiple input languages, switch to English (hopefully United States).
  3. Open Notepad or other programs with an edit field.
  4. Type something.

Modern IME (mostly Korean)

  1. Install a langue that comes with IME (such as Korean display langauge).
  2. Type something in Korean or other languages.

Actual behavior:

NVDA does not announce top input suggestions including IME candidates when IME window opens.

Expected behavior:

NVDA announces top suggestion for hardware keyboard input suggestions and IME entries when IME window opens for some languages.

NVDA logs, crash dumps and other attachments:

None

System configuration

NVDA installed/portable/running from source:

Installed

NVDA version:

alpha-31360,dd44cbb7

Windows version:

Windows 11 23H2 beta (22635.3286)

Name and version of other software in use when reproducing the issue:

None

Other information about your system:

Korean display language and input method editor (IME) installed

Other questions

Does the issue still occur after restarting your computer?

Yes

Have you tried any other versions of NVDA? If so, please report their behaviors.

Same behavior in 2023.3.x and 2024.1 beta

If NVDA add-ons are disabled, is your problem still occurring?

Yes

Does the issue still occur after you run the COM Registration Fixing Tool in NVDA's tools menu?

Not applicable

Cause and solution

The UIA ImeCandidateUI's show event handler (found in modern keyboard/emoji panel app module) knows that "Ime_PredictionWindow" class is used for a variety of things and will invoke UIA element selected event for its immediate first child. However, in Windows 11, there is something else underneath the prediction window, so NVDA must move one more level down. This will then allow hardware keyboard input suggestion item to be announced when the user types. As for IME candidate announcement while opening IME window, this can be resolved by relaxing Windows 11 check in ImeCandidateItem's UIA element selected event to veto the event if focused on the IME candidate window (both element selected and gain focus events are fired by IME candidate items). A partial fix is included in Windows App Essentials ad-on, with a pull request to resolve it from NVDA Core is on its way.

Thanks.
@seanbudd seanbudd added p3 https://github.com/nvaccess/nvda/blob/master/projectDocs/issues/triage.md#priority feature/i18n Internationalization features triaged Has been triaged, issue is waiting for implementation. labels Mar 11, 2024
josephsl pushed a commit to josephsl/nvda that referenced this issue Mar 15, 2024
Windows 11/hardware keyboard input suggestions: descend one more leve…
4cb306b 
…l when handling show event from IME candidate UI if Automation Id is 'IME_Prediction'Window'. Re nvaccess#16283.

When show event is handled in candidate UI, descend one more level if UIA Automation Id is 'IME_PredictionWindow' a.k.a. hardware keyboard input suggestions. This allows hardware keyboard input suggestions item in Windows 11 to handle element selected event (with some refinements).
josephsl pushed a commit to josephsl/nvda that referenced this issue Mar 15, 2024
Modern keyboard/candidate item: do handle element selected even if no…
3e057c9 
…t focused on candidate UI. Re nvaccess#16283.

Windows 11 hardware keyboard input suggestions: announce top suggestion if keyboard input suggestion is enabled just as it does in Windows 10. This is done by refining UIA element selected event in IME candidate item class - instead of vetoing it in Windows 11, it will be vetoed if gain focus event is pending/parent element is candidates list.
josephsl pushed a commit to josephsl/nvda that referenced this issue Mar 15, 2024
What's new: add bug fix entry on Windows 11 hardware keyboard suggest…
3e880c7 
…ion announcement. Re nvaccess#16283
This was referenced Mar 15, 2024
Merged
Closed
@josephsl
Copy link
Collaborator Author

Hi,

Update: for announcing visible IME candidates, a separate pull request will be made available as it requires handling focus entered event. The case of element selected event will handle IME candidates announcement in some cases.

Thanks.

josephsl pushed a commit to josephsl/nvda that referenced this issue Mar 30, 2024
@josephsl
Windows 11/hardware keyboard input suggestions: descend one more leve…
eb45465 
…l when handling show event from IME candidate UI if Automation Id is 'IME_Prediction'Window'. Re nvaccess#16283.

When show event is handled in candidate UI, descend one more level if UIA Automation Id is 'IME_PredictionWindow' a.k.a. hardware keyboard input suggestions. This allows hardware keyboard input suggestions item in Windows 11 to handle element selected event (with some refinements).
josephsl pushed a commit to josephsl/nvda that referenced this issue Mar 30, 2024
@josephsl
Modern keyboard/candidate item: do handle element selected even if no…
fda6078 
…t focused on candidate UI. Re nvaccess#16283.

Windows 11 hardware keyboard input suggestions: announce top suggestion if keyboard input suggestion is enabled just as it does in Windows 10. This is done by refining UIA element selected event in IME candidate item class - instead of vetoing it in Windows 11, it will be vetoed if gain focus event is pending/parent element is candidates list.
josephsl pushed a commit to josephsl/nvda that referenced this issue Mar 30, 2024
@josephsl
What's new: add bug fix entry on Windows 11 hardware keyboard suggest…
7f50613 
…ion announcement. Re nvaccess#16283
josephsl added a commit to josephsl/nvda that referenced this issue Mar 30, 2024
@josephsl
Emoji panel/element selected: remove Windows 11 IME window Automation…
1908183 
… Id check as it is now dead code. Re nvaccess#16283.

Now that IME candidate UI knows where candidate items are in Windows 11, there is no need to do redundant checks in app module version of element selected event handler, making it dead code (and thus removed).
michaelDCurran pushed a commit that referenced this issue Apr 1, 2024
@josephsl
Windows 11/modern keyboard: handle hardware keyboard input suggestion…
0a55ad2 
…s announcements again (#16313)

Closes #16283

Summary of the issue:
In Windows 11, NVDA does not announce top suggestion from hardware keyboard input suggestions list.

Description of user facing changes
NVDA will once again announce hardware keyboard input suggestions.

Description of development approach
Edited IMECandidateUI/IMECandidateItem classes in emoji panel app module:
• IMECandidateUI: in Windows 11, NVDA will descend one more level if Automation Id is "IME_Prediction_Window".
• IMECandidateItem: NVDA will handle UIA element selected event as long as focus is not on the candidate item itself (refined from previous iteratino of this class).
@nvaccessAuto nvaccessAuto added this to the 2024.2 milestone Apr 1, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
feature/i18n Internationalization features p3 https://github.com/nvaccess/nvda/blob/master/projectDocs/issues/triage.md#priority triaged Has been triaged, issue is waiting for implementation.
Projects
None yet
Milestone
2024.2
3 participants
@seanbudd @josephsl @nvaccessAuto