Windows 11 Design Elements Are Not Reported by NVDA When Mouse Tracking Is On

[ferhatbkr]

Steps to reproduce:

• Turn on mouse tracking in NVDA.
• Hover mouse cursor over taskbar items. In the same way, navigate the toolbar buttons of Windows Explorer, the top menus of the New Notepad application with the mouse.

Actual behavior:

NVDA will just say "DesktopWindowXamlSource" instead of naming items.
The same is true for all applications developed with Windows 11's new Flat Design. Like PowerToys that don't come pre-installed with Windows.
Narrator items read their names as expected.

Expected behavior:

NVDA must accurately report information on items, as Narrator does.

System configuration

NVDA installed/portable/running from source:

Installed

NVDA version:

NVDA 2021.3.4

Windows version:

Windows 11 Version 10.0.22000.588 Build 22000.588

Name and version of other software in use when reproducing the issue:

Other information about your system:

Other questions

Does the issue still occur after restarting your computer?

Yes

Have you tried any other versions of NVDA? If so, please report their behaviors.

If NVDA add-ons are disabled, is your problem still occurring?

Yes

Does the issue still occur after you run the COM Registration Fixing Tool in NVDA's tools menu?

Yes

[josephsl]

Hi,

Can you tell us things Narrator says so we can investigate what's going on?

Technical: these design elements expose class names as their label (name).

Thanks.

[josephsl]

Hi,

Also, note that design elements are not really the same thing as item labels, and I think that could confuse people.

Thanks.

[Molitvan]

I have the same problem. Also, I noticed that its happening with almost all new ms apps like windows terminal for example. I assume that some library used in these apps/elements is making an issue.

Also, I think there is a way to fix the problem with taskbar. Actualy, in windows 11, there is some kind of a cover on top of the taskbar. This cover makes nvda read that instead of the actual taskbar when you hover over it. You can test this by simply hovering over the top edge of the taskbar. If there is a way to remove this, the taskbar will work. Also, by removing that cover you will restore the old taskbar context menus :D.

I will update this comment if my theory about the taskbar turns out to be true

[ferhatbkr]

Hi,

Can you tell us things Narrator says so we can investigate what's going on?

Technical: these design elements expose class names as their label (name).

Thanks.

The name or tooltip of the Start Menu, pinned applications, and system tray items (including date and time) in the Taskbar is not reported by NVDA.

In Windows Explorer, buttons and menus on the bar that Narrator calls "Application Bar" (ex: new, cut, copy, rename buttons and sort, view menus) are not spoken.

I said "design element" because I had this problem in applications updated according to this design, which came with the new design of Widnows 11. My fault.

[josephsl]

Hi,

About terminology: yep, I can see why it can be confusing.

Also another technical note: until Windows 10 at least shell elements such as Start button label was exosed as a MSAA (Microsoft Active Accessibility) objects, whereas they are seen as UIA (UI Automation) objects in Windows 11.

Thanks.

[josephsl]

Hi,

Dramatic turn: this explains why touch hover gesture (hold down one finger while navigating the screen) is not effective in Windows 11 taskbar - no item announcement when one hovers around the bottom of the screen. I will raise a separate issue for touchscreen interaction as looks like these might be related.

Thanks.

[josephsl]

Hi,

April 2022 update: telling NVDA to treat DesktopWindowXamlSource objects as UIA fixes it for BOTH mouse and touch interaction. Note that there are multiple elements with this "label", so the easy fix will require informing the File Explore app module that these elements are indeed UIA objects. I'll come up with a prototype fix via Windows App Essentials add-on if possible (possibly later today which will require changes to File Explorer app module).

Thanks.

[josephsl]

Hi,

Update: yes, UIA element reclassification fixes it. For folks wishing to write a pull request (I can do so next week if needed), the following window class names must be checked from isGoodUIAWindow method in File Explorer app module class: MSTaskSwWClass (parts of taskbar), TrayNotifyWnd (systray list), ToolbarWindow32 (toolbar), ReBarWindow32 (parts of taskbar), Shell_TrayWnd (taskbar). Of course you must make sure this is Windows 11 or later. I'm sure there is a much better solution.

Thanks.

[josephsl]

Hi,

For Windows 11 Notepad, a window with the class name of "Notepad" shuold be treated as good UIA window. Because of this, a solution that works across programs is ideal.

Additional findings:

1. DesktopWindowXamlSource class name is used by classic Win32 apps to host XAML controls.
2. A more effective solution is detecting these windows from UIA handler thread itself and telling NVDA to treat these as good UIA windows. While editing isGoodUIAWindow method from app modules would work, it isn't generalizable across apps as the class name in question (see above) is found across multiple apps.
3. No separate issue is required to describe touchscreen interaction behavior as the issue described here impacts both mouse and touch. Therefore a pull request that fixes this problem will affect both interaction scenarios at once.
4. As a workaround, enable "report tooltips" from object presentation settings panel and enable "follow mouse cursor" checkbox from review cursor settings panel.
5. Although the computer used to test possible solutions is running a nickel build (22598 to be exact), a pull request for this issue will also work on Windows 11 Version 21H2 (build 22000) as evidenced by the fact that the described workaround and potential solutions are working in a virtual machine running the said build.

As for pull requests, while I may not be able to come up with a solution that works across apps (I'm exhausted), I can do something about at least File Explorer (Windows 11 Notepad will be handled via Windows App Essentials for now).

CC @michaelDCurran in case you heard something from XAML team inside Microsoft.

Thanks.