UIA handler: introduce UIA drag/drop event handlers for use with announcing items that were rearranged on screen #12271
Labels
api/uia
Feature or bug requires an understanding of UIA
app/windows-interface
Interactions between NVDA and the default Windows GUI
p4
https://github.com/nvaccess/nvda/blob/master/projectDocs/issues/triage.md#priority
triaged
Has been triaged, issue is waiting for implementation.
Milestone
Comments
|
Hi, Actually, gain focus event won't work directly as NVDA (through UIA handler thread) says focus event was ignored by the dragged object. So it appears dedicated events should be created in UIA objects to queue gain focus event for focused objects. Thanks. |
|
This would be very helpful for the highlighter as well. |
josephsl
added a commit
to josephsl/nvda
that referenced
this issue
May 10, 2021
…. Re nvaccess#12271. UIA drag complete and drop target dropped events are used to announce new location for an element. The former is seen when rearranging tiles in Windows 8 start screen, and the latter is triggered when arranging quick actions in Windows 10 Action Center. Although both events might as well call gain focus event handler, specific events will be defined in UIA objects for future-proofing.
josephsl
added a commit
to josephsl/nvda
that referenced
this issue
May 10, 2021
When tiles are rearranged in Windows 8 start screen, braille reports new location whereas speech doesn't. Therefore use UIA drag complete event to force focus to be reported.
josephsl
added a commit
to josephsl/nvda
that referenced
this issue
May 10, 2021
. Unlike UIA drag complete event, drop target dropped event is raised by a containing element where items are being rearragned, the best example being rearranging quick actions in Windows 10 Action Center. At least NVDA does know what the focused element is, therefore raise a gain focus event on the focused object so the newly dragged item can be announced.
7 tasks
josephsl
added a commit
to josephsl/nvda
that referenced
this issue
May 18, 2021
…ions with Flake8 F405 NOQA (definition). Re nvaccess#12271
josephsl
added a commit
to josephsl/nvda
that referenced
this issue
May 19, 2021
…vaccess#12271. Pointed out by Leonard de Ruijter: obj.reportFocus does not allow vision enhancmenet providers and other components from reacting to gain focus event. Therefore call obj.event_gainFocus when drag complete and/or drop target dropped events are fired. Also, add a source code comment on drop target dropped event that points out which element fires this event.
josephsl
added a commit
to josephsl/nvda
that referenced
this issue
May 22, 2021
…s (integers). Re nvaccess#12271. Reminder from Leonard de Ruijter: reference UIA module when defining UIA event constants (UIA.constant).
|
Adding this comment here instead of on the closed PR #12428:
Originally posted by @josephsl in #12428 (comment) |
|
I guess to get the work going, #12428 can be split up in parts so that at least the basic reporting of states is there. |
|
Hi, in this case, at least two PR’s will be created: one to handle drag complete event, the other to handle drop target event. A possible third PR might be introduced to handle drag drop effect property which might be the easiest to handle. Thanks.
|
|
Hi, Based on code examiantion, looks like at least four PR's are needed:
I can come up with these PR's within a week. Thanks. |
seanbudd
added
api/uia
This was referenced Aug 29, 2022
josephsl
added a commit
to josephsl/nvda
that referenced
this issue
Sep 2, 2022
…ss#12271. Revisiting NVDA issue 12271: UIA elements can provide drag drop effect, a text describing where the dragged object is located in relation to other elements. A somewhat related property, called drop target drop target effect property, is used by elements supporting drop target events to describe what happened when elements were dragged and dropped (rearranged in some cases). Most notably, the former property changes when rearranging Windows 10 Start menu tiles, and the latter property changes when rearranging virtual desktops in Windows 11. As a first step, recognize drag drop effect property changes, with the actual implementation coming in the next commit.
josephsl
added a commit
to josephsl/nvda
that referenced
this issue
Sep 2, 2022
…#12271. The other side of drag and drop is drop target events set and their effects on the item being dragged and/or drop target element. For drop target effect, it is the element to which the dragged item can be dropped into, meaning it affects a different element other than the item being dragged at the moment. Therefore let NVDA anounce drop target effect property changes if given a chance to do so (implementation is the next commit).
josephsl
added a commit
to josephsl/nvda
that referenced
this issue
Sep 2, 2022
…ess#12271. Handle drop target effect property changes, hopefully from the object that actually is the target of a drag and drop operation. However there are cases where it is not the actual drop target element that records the effect property, notably when rearranging quick settings items in Windows 11. Therefore traverse focus ancestors until an element with drop target effect property is found and then report it as the effect property. The implication of this corner case is that drop target property change must be fetched globally (next commit).
josephsl
added a commit
to josephsl/nvda
that referenced
this issue
Sep 2, 2022
…erty change event. Re nvaccess#12271. Because drop target effect property can come from an element that is not necessarily the immediate focus element, treat it as though it is a global event. This is more so in Windows 11 Version 22H2 where selective UIA event registration is the default and drop target effect changes when rearranging quick settings items in Windows 11.
6 tasks
josephsl
added a commit
to josephsl/nvda
that referenced
this issue
Sep 4, 2022
…access#12271. Because a drop target element can tell UIA that drop target effect can change many times, it becomes hard to notify users about latest happenings iwth drag and drop operation. Therefore cancel speech every time drop target effect property changes, similar to layout invalidated event where suggestions count changes multiple times.
josephsl
added a commit
to josephsl/nvda
that referenced
this issue
Sep 4, 2022
…s. Re nvaccess#12271." This reverts commit cd8f1d0 as it causes 'dragging' to not be announced when using the keyboard to rearrange tiles in Windows 10. It still causes effects to be announced continuously when using a mouse, so a different approach for dealing with input sources other than keyboards may need to be investigated.
seanbudd
added a commit
that referenced
this issue
Sep 6, 2022
Fixes #12271 Follow-up and a different approach to #12428 Summary of the issue: When drag and drop target events happen, NVDA does not announce their effects such as screen content changes such as effect of rearranging Start menu tiles. Description of user facing changes NVDA will announce effects of drag and drop events in places such as: - Windows 10 Start menu: after moving tiles - Windows 10 Action center/Windows 11 quick setings: afterreordering settings items - Windows 11 task view: rearranging virtual desktops - Windows 11 Start menu: rearranging pinned items, and from Version 22H2 onwards, creating app folders Description of development approach While this PR may look similar to #12428, the internals are different: - In UIA handler, added property events for drag drop and drop target effects. - In UIA objects, add property events for drag drop and drop target effects and announce effect text. - For drop target effect, because drop target event may come from somewhere other than the object that "raises" it, traverse focus ancestors looking for UIA elements with drop target effect text set. - Back in UIA handler, because drop target effect event can come from somewhere other than the focused element or its parent, treat it as a global event. --- * UIA handler: introduce UIA drag drop effect property event. Re #12271. Revisiting NVDA issue 12271: UIA elements can provide drag drop effect, a text describing where the dragged object is located in relation to other elements. A somewhat related property, called drop target drop target effect property, is used by elements supporting drop target events to describe what happened when elements were dragged and dropped (rearranged in some cases). Most notably, the former property changes when rearranging Windows 10 Start menu tiles, and the latter property changes when rearranging virtual desktops in Windows 11. As a first step, recognize drag drop effect property changes, with the actual implementation coming in the next commit. * NVDAObjects.UIA: handle UIA drag drop effect property. Re #12271. Announce drag drop effect if it can be fetched (introduced in Windows 8). * UIA handler: introduce drop target effect property event. Re #12271. The other side of drag and drop is drop target events set and their effects on the item being dragged and/or drop target element. For drop target effect, it is the element to which the dragged item can be dropped into, meaning it affects a different element other than the item being dragged at the moment. Therefore let NVDA anounce drop target effect property changes if given a chance to do so (implementation is the next commit). * NVDAObjects.UIA: handle drop target effect property changes. Re #12271. Handle drop target effect property changes, hopefully from the object that actually is the target of a drag and drop operation. However there are cases where it is not the actual drop target element that records the effect property, notably when rearranging quick settings items in Windows 11. Therefore traverse focus ancestors until an element with drop target effect property is found and then report it as the effect property. The implication of this corner case is that drop target property change must be fetched globally (next commit). * UIA handler: treat drop target effect property event as a global property change event. Re #12271. Because drop target effect property can come from an element that is not necessarily the immediate focus element, treat it as though it is a global event. This is more so in Windows 11 Version 22H2 where selective UIA event registration is the default and drop target effect changes when rearranging quick settings items in Windows 11. * UIA handler/property names to NVDA events: lint * NVDAObjects.UIA: cancel speech when drop target effect changes. Re #12271. Because a drop target element can tell UIA that drop target effect can change many times, it becomes hard to notify users about latest happenings iwth drag and drop operation. Therefore cancel speech every time drop target effect property changes, similar to layout invalidated event where suggestions count changes multiple times. * Revert "NVDAObjects.UIA: cancel speech when drop target effect changes. Re #12271." This reverts commit cd8f1d0 as it causes 'dragging' to not be announced when using the keyboard to rearrange tiles in Windows 10. It still causes effects to be announced continuously when using a mouse, so a different approach for dealing with input sources other than keyboards may need to be investigated. * UIA handler: add comma for ease of diff comparisons * update changes * NVDAObjects.UIA (optimization): do not fetch UIA property cache if this is not a UIA object Suggested by Leonard de Ruijter: just continue the ancestor retrieval loop if NVDA came across a non-UIA object as it avoids handling COM error. Co-authored-by: Leonard de Ruijter <leonardder@users.noreply.github.com> * NVDAObjects.UIA: remove unneeded attribute error check. Suggested by Leonard de Ruijter: now that only COM error is handled, there is no need to handle attribute error as cacheable value will not be fetched for non-UIA objects. Co-authored-by: Leonard de Ruijter <leonardder@users.noreply.github.com> * NVDAObjects.UIA: NVDAObjects.UIA.UIA -> UIA when checking instance. Co-authored-by: Sean Budd <sean@nvaccess.org> Co-authored-by: Leonard de Ruijter <leonardder@users.noreply.github.com>
josephsl
added a commit
to josephsl/nvda
that referenced
this issue
Oct 21, 2022
…vent. Re nvaccess#12271. Follow-up/quick fix engineering for drag and drop effect announcement: suppose a user moves the mouse to an object that: is a UIA element, supports drag and drop, and is not a system focus. When drag and drop operation starts, drag drop effect is not announced, caused by the drag drop effect not being a global property event. Beta branch justification: this is a bug fix/new feature introduced in NVDA 2022.4. Subsequent testing indicate that drag drop UIA property should have been a global property so the property can be announced based on mouse actions, simiar to reporting mouse when mouse tracking is turned on via mouse settings panel. This change also aligns with drop target effect announcement as drag and drop announcement issue covers both.
6 tasks
seanbudd
pushed a commit
that referenced
this issue
Oct 23, 2022
…vent. Re #12271. (#14278) Link to issue number: Follow-up to #12271 Follow-up to #14081 Summary of the issue: Follow-up/quick fix engineering for drag and drop effect announcement: suppose a user moves the mouse to an object that: is a UIA element, supports drag and drop, and is not a system focus. When drag and drop operation starts, drag drop effect is not announced, caused by the drag drop effect not being a global property event. Description of user facing changes Drag drop effect will be announced when performing drag and drop operation via mouse regardless of system focus, similar to drop target effect announcement. Description of development approach Add UIA_DragDropEffectPropertyId to global properties list to align with drop target effect property. Follow-up/quick fix engineering for drag and drop effect announcement: suppose a user moves the mouse to an object that: is a UIA element, supports drag and drop, and is not a system focus. When drag and drop operation starts, drag drop effect is not announced, caused by the drag drop effect not being a global property event. Beta branch justification: this is a bug fix/new feature introduced in NVDA 2022.4. Subsequent testing indicate that drag drop UIA property should have been a global property so the property can be announced based on mouse actions, simiar to reporting mouse when mouse tracking is turned on via mouse settings panel. This change also aligns with drop target effect announcement as drag and drop announcement issue covers both.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Hi,
I believe the following was asked before, but figured it would be best to put it in a systematic way:
Background:
When rearranging UIA objects such as Start menu tiles and Action Center items (Windows 10), UIA can raise drag drop events. This is checked by Narrator (at least) to announce the new object location. For example, when rearranging Action Center items by pressing Alt+Shift+left/right arrow keys, screen readers may announce the new location of the just dragged object. This feature will become useful in future Windows 10 releases as latest Windows Insider Preview build allows one to rearragne virtual desktops and UIA drag drop events are fired.
Is your feature request related to a problem? Please describe.
Currently NVDA does not announce items or new locations when items are rearranged. Technially, it does through windows 10 app Essentials add-on, but it would be better to bring this to core.
Describe the solution you'd like
Let NVDA listen to UIA drag/drop events and announce the rearrange item and new location by calling obj.reportFocus method.
Describe alternatives you've considered
Leave it up to add-ons to do this.
Additional context
For a practical example and justifications, see:
https://blogs.windows.com/windows-insider/2021/03/17/announcing-windows-10-insider-preview-build-21337/
Implementation strategy:
UIA drag/drop event handler code from Windows 10 App Essentials add-on will be ported to NVDA Core and will correspond to gain focus event unless folks believe a separate set of UIA events should be created. There are six events related to drag and drop, but the events we are interested in are "drag dropped" and "drag complete".
Thanks.
The text was updated successfully, but these errors were encountered: