NVDAObjects.UIA: let NVDA recognize drag and drop states based on IsGrabbed property

[josephsl]

Hi,

This issue stems from #12271

Is your feature request related to a problem? Please describe.

NVDA objects representing UIA elements can expose UIA patterns through dev info facility (NVDA+F1). While some patterns are mapped to object states, a number of useful patterns to states mapping are missing.

A prominent group of patterns relate to drag and drop. Specificaly, UIA elements can expose drag and drop target patterns for use by assistive technologies to inform users about draggable items. This is used in Narrator and NVDA (the latter via an add-on) to notify users about drag and drop target operations such as rearranging virtual desktops on Windows 11, placing Start menu times in Windows 10, and managing apps and folders in Windows 11 Version 22H2 Start menu. While these patterns are exposed by elements, they are not mapped to drag and drop target states yet.

Describe the solution you'd like

Since NVDA does come with drag and drop target states in control types package, let UIA NVDA objects add drag/drop target states based on exposed patterns.

Describe alternatives you've considered

None - cannot be done easily via add-ons as it touches NVDAObjects.UIA.UIA._get_states getter.

Additional context

This issue came from discussions surrouding #12271 as the pull request for it was unmanageable and led to user experience issues. Therefore, this issue serves as a starting point for resolving the mentioned issue in a series of pull requests to make the work more manageable.

Tanks.

[josephsl]

People who might be interested: @LeonarddeR, @codeofdusk, other UIA users. ONly after introducing drag/drop patterns to object states can a reimplementation of #12271 commence (I think it would make sense to not mess with gain focus events anymore as I have learned some things over the years that makes focus event queueing not worth it now).

[josephsl]

Hi,

Actualy, a simpler approach exists: UIA_IsGrabbedPropertyId (issue title changed).

Thanks.

[mzanm]

I'm working on this and one issue I'm encountering is that IsGrabbed property only changes when dragging using the mouse, using the keyboard doesn't change it.

[josephsl]

Hi, there is one place where you can test this with keyboards: Windows 10 Start menu tiles, specifically when rearranging tiles by pressing Alt+Shift+arrow keys. I have a working implementation, but I can wait a while (my implementation involves listening to drag start UIA event). Thanks.

[mzanm]

Yes, that's what I'm doing, it gets dragged and dropped but no grabbing property while holding alt + shift like how when the left button on the mouse is held down over an item.

[josephsl]

Hi,

Interesting. The way I resolved this in Windows Ap Essentials add-on (privately testing in canary channel a.k.a. source code level) is to map UIA drag start event to state change event, then in state change event inside the global plugin part of the add-on, check if is grabbed property is set and add "dragging" state to obj.states set. By the time UIA object gets to handle state change event (nextHandler function), NVDA will announce "dragging". But this is only one part of the puzzle: the other part, to be raised in a separate issue and will be addressed at the same time as this one, involves listening to UIA drag drop effect property changes and announcing the progress of drag drop process (same as Narrator). There is one last component which might be addressed in a separate PR: letting NVDA announce "done dragging" when drag and drop completes (this involves listening to UIA drag complete event). This whole algorithm responds to drag drop events set - there is another algorithm involved in listening to drop target events, which is not exacly the same thing as drag drop events set but somewhat related (that one will (finally) address NVDA not announcing placement progress when rearranging a variety of things, most notably Windows 10/11 quick actions). All these will be packaged as part of August 31st dev build of Windows App Essentials and may show up as part of 22.10 to stable channel subscribers in a few weeks.

So I propose that we compare our notes and see if we can come up with a common implementation (I think it would be better (in the long term) for me to step aside).

Thanks.

[josephsl]

Hi,

As for translating this into NVDA Core, this will involve editing at least two files: UIAHadler/init.py, and NVDAObjects/UIA/init.py. The former will add a few events and properties to UIA events/properties to NVDA events maps, and the latter will handle "is grabbed" property and map it to object states (NVDAObjects.UIA.UIA._get_states).

Thanks.

[mzanm]

The way I tried to resolve this is I mapped the property directly to stateChanged
Is there a reason it wouldn't work that way?

[josephsl]

Hi, it will work if drag start event is also mapped to state change event. Thanks.

[mzanm]

Just tried that, state is announced with mouse, with keyboard it is grabbed but nothing.
I've also noticed that a pane that is the parent of the taskbar pinned icons is sending the drag start event, so it might be caused by that as it's not focused.

[josephsl]

Hi, okay, just to make sure we are on the same page: are you talking about rearranging taskbar icons in Windows 11, or rearranging Start menu tiles in Windows 10? If the former, then I’m afraid that the latest dev build of Windows App Essentials add-on might not help; if the latter, then the latest add-on dev build will address it. The implementation I described is part of latest Windows App Essentials add-on development build (August 31st build, that is) – not officially committed to Git repo as I want to test the idea before officially committing to it (which I might do once I feel better from the latest seasonal headache). Thanks.

[mzanm]

I've just tried latest windows app essentials addon from git and it has the same issue as my implementation.
Is it supposed to announce the dragging state when? I'm on windows 11 22 h 2 and I also tried on 21 h 2 before I upgraded and so far, I have tried rearranging taskbar icons, pinned tiles, virtual desktops, the dragging state does not get reported.
Edit: Does it only work on 10?

[josephsl]

Hi,

Update: a follow-up pull request is on its way - adding and mapping UIA drag cancel and complete events to state changes.

[josephsl]

Hi, it will announce dragging state from keyboard if the element supports “is grabbed” property, the best place to test it being Windows 10 Start menu – I haven’t found a place in Windows 11 where it works properly. Activities such as rearranging quick settings items will yield drop target events instead, not drag start/cancel/complete events, hence our work only addresses part of the issue. I’m about to publish a pull request, and I’ll mention your ID there and let me know if that’s the implementation you’ve got, in which case we might as well combine our thought processes. Thanks.