Windows 11/hardware input suggestions and IME: NVDA does not announce top hardware keyboard input suggestions or candidates when IME window opens

[josephsl]

Hi,

Stems from #14509:

Background

With #14509, NVDA will not announce IME entries twice when modern IME opens in Windows 11. Unfortunately, this did not cover the case of hardware keyboard input suggestions where the top result was annoucned in Windows 10 whereas it does not in Windows 11, traced to IME window traversal due to UIA tree differences in modern keyboard/IME window.

Steps to reproduce:

Hardware keyboard input suggestions

1. Enable hardware keyboard input suggestions in Windows 11 (Settings/Time and language/Typing, enable "Show text suggestions when typing on the physical keyboard").
2. If using multiple input languages, switch to English (hopefully United States).
3. Open Notepad or other programs with an edit field.
4. Type something.

Modern IME (mostly Korean)

1. Install a langue that comes with IME (such as Korean display langauge).
2. Type something in Korean or other languages.

Actual behavior:

NVDA does not announce top input suggestions including IME candidates when IME window opens.

Expected behavior:

NVDA announces top suggestion for hardware keyboard input suggestions and IME entries when IME window opens for some languages.

NVDA logs, crash dumps and other attachments:

None

System configuration

NVDA installed/portable/running from source:

Installed

NVDA version:

alpha-31360,dd44cbb7

Windows version:

Windows 11 23H2 beta (22635.3286)

Name and version of other software in use when reproducing the issue:

None

Other information about your system:

Korean display language and input method editor (IME) installed

Other questions

Does the issue still occur after restarting your computer?

Yes

Have you tried any other versions of NVDA? If so, please report their behaviors.

Same behavior in 2023.3.x and 2024.1 beta

If NVDA add-ons are disabled, is your problem still occurring?

Yes

Does the issue still occur after you run the COM Registration Fixing Tool in NVDA's tools menu?

Not applicable

Cause and solution

The UIA ImeCandidateUI's show event handler (found in modern keyboard/emoji panel app module) knows that "Ime_PredictionWindow" class is used for a variety of things and will invoke UIA element selected event for its immediate first child. However, in Windows 11, there is something else underneath the prediction window, so NVDA must move one more level down. This will then allow hardware keyboard input suggestion item to be announced when the user types. As for IME candidate announcement while opening IME window, this can be resolved by relaxing Windows 11 check in ImeCandidateItem's UIA element selected event to veto the event if focused on the IME candidate window (both element selected and gain focus events are fired by IME candidate items). A partial fix is included in Windows App Essentials ad-on, with a pull request to resolve it from NVDA Core is on its way.

Thanks.

[josephsl]

Hi,

Update: for announcing visible IME candidates, a separate pull request will be made available as it requires handling focus entered event. The case of element selected event will handle IME candidates announcement in some cases.

Thanks.