[wpf] Add UIA events #14097

carlos-zamora · 2022-09-28T17:04:38Z

Adds UIA events to the WPF control for the following items:

selection changed
text changed (and output)
cursor changed

Automation Peer

Similar to the architecture of the UWP TermControl, we added a
HwndTerminalAutomationPeer which acts as the
TermControlAutomationPeer in UWP. However, we don't need a XAML
wrapper here, so really we just need it to inherit from
TermControlUiaProvider (the ITextProvider implementation shared
across conhost and WT) and IUiaEventDispatcher (the event dispatching
interface that is responsible for signaling the screen reader that
something has changed).

Removing the local echo

As with WT, we need to record key events to remove the local echo. These
recorded events are matched up with the output text. Each sequential
match is removed in the output text so that it's not read by the screen
reader.

Detecting what to send events for

As with WT, a UiaEngine was added to the renderer and it is set up
when a UIA client is detected. WT would normally stop sending events
when focus was lost from the control. We do the same here.

Automation properties

TermControlUiaProvider was upgraded to support property values. Such
properties include class name and control type. These align with those
set in TermControlAutomationPeer. Realistically, those should point to
these, but that requires a lot more work and a localization burden
(because we need to move the localized word "terminal").

HwndTerminalAutomationPeer takes this a step further and overrides the
class name to be WPFTermControl. This allows screen readers to provide
special handling for the WPFTermControl vs the UWP term control since
they will be updating at different speeds.

Build fixes

To build the WPF test app, I had to mess with the dependencies a little
bit. Really just add the atlas engine and uia renderer to the build
steps.

HwndTerminal initialization

The initialization order with WM_NCCREATE was changed to match that of
Windows Terminal (BaseWindow/IslandWindow). This is safer now. I also
removed the static window because it was unnecessary.

Handling `WM_GETOBJECT`

WPF's HwndHost likes to mark the WM_GETOBJECT message as handled to
force the usage of the WPF automation peer. We now explicitly mark it as
not handled and don't return an automation peer. This forces the message
to go down to the HwndTerminal where we return terminal's UiaProvider.

Remove TermControl layer from UIA tree

TermContol (the top-most layer in the UIA tree) would pop up and not do
anything. This PR also overrides the automation peer at that layer and
marks IsContentElement/IsControlElement=false (the equivalent to
AccessibilityView=Raw). This makes the layer only appear in the UIA tree
if you are using the raw view (i.e. you know what you're doing and you
want to see each individual layer even if you can't directly interact
with it).

Validation Steps Performed

Tested with Narrator/NVDA using WpfTerminalTestNetCore project in our
repo.

New output is read out (not just key events, but also other output
text)
Local echo does not occur (i.e. pressing 'A' should only read 'A'
once, not twice [key event and rendered letter]).
selection events are read out properly
cursor change events are read out properly (tested with text
cursor indicator preview in Settings App > Accessibility > Text
Cursor)

NOTE: test this with Release builds. Debug builds may be too slow and
not read out properly

Closes #12642

src/cascadia/PublicTerminalCore/HwndTerminalAutomationPeer.cpp

src/cascadia/PublicTerminalCore/HwndTerminalAutomationPeer.hpp

src/cascadia/PublicTerminalCore/HwndTerminal.cpp

codeofdusk · 2022-09-28T17:46:25Z

Does this new control use the same UIA class name? How can I test it?

src/types/TermControlUiaProvider.cpp

- remove extra static window - improve window initialization order - pass WM_GETOBJECT to HwndTerminal - set accessibility view to raw on TermControl - rename UIA class name to WPFTermControl - fix build for WpfTerminalTestNetCore (Debug and Release)

src/cascadia/PublicTerminalCore/HwndTerminal.cpp

lhecker · 2022-10-06T21:39:34Z

src/cascadia/PublicTerminalCore/HwndTerminal.cpp

+    if (WM_NCCREATE == uMsg)
+    {
+#pragma warning(suppress : 26490) // Win32 APIs can only store void*, have to use reinterpret_cast
+        auto cs = reinterpret_cast<CREATESTRUCT*>(lParam);


BTW you can now use std::bit_cast for this. But it's not particularly important.

lhecker · 2022-10-06T21:42:58Z

src/cascadia/PublicTerminalCore/HwndTerminalAutomationPeer.cpp

+            // the key event's character (i.e. the "A" key) matches
+            // the output character (i.e. "a" or "A" text).
+            // We can assume that the output character resulted from
+            // the pressed key, so we can ignore it.


There's no way this works with Unicode, right? ._.

Probably not, but that should be fine. We're really just using this to suppress key events. So, when you press the A key, normally you'd hear "A" from pressing the key and "A" from rendering the key. If we suppress that second event, it sounds/feels more natural.

src/types/TermControlUiaProvider.cpp

src/cascadia/PublicTerminalCore/HwndTerminal.cpp

DHowett

memory questions!

src/cascadia/PublicTerminalCore/HwndTerminal.cpp

src/cascadia/PublicTerminalCore/HwndTerminalAutomationPeer.cpp

src/cascadia/WpfTerminalControl/TerminalContainer.cs

src/cascadia/PublicTerminalCore/HwndTerminalAutomationPeer.cpp

ghost · 2022-10-06T22:40:32Z

Hello @DHowett!

Because this pull request has the AutoMerge label, I will be glad to assist with helping to merge this pull request once all check-in policies pass.

p.s. you can customize the way I help with merging this pull request, such as holding this pull request until a specific person approves. Simply @mention me (`@msftbot`) and give me an instruction to get started! Learn more here.

Adds UIA events to the WPF control for the following items: - selection changed - text changed (and output) - cursor changed Similar to the architecture of the UWP TermControl, we added a `HwndTerminalAutomationPeer` which acts as the `TermControlAutomationPeer` in UWP. However, we don't need a XAML wrapper here, so really we just need it to inherit from `TermControlUiaProvider` (the `ITextProvider` implementation shared across conhost and WT) and `IUiaEventDispatcher` (the event dispatching interface that is responsible for signaling the screen reader that something has changed). As with WT, we need to record key events to remove the local echo. These recorded events are matched up with the output text. Each sequential match is removed in the output text so that it's not read by the screen reader. As with WT, a `UiaEngine` was added to the renderer and it is set up when a UIA client is detected. WT would normally stop sending events when focus was lost from the control. We do the same here. `TermControlUiaProvider` was upgraded to support property values. Such properties include class name and control type. These align with those set in `TermControlAutomationPeer`. Realistically, those should point to these, but that requires a lot more work and a localization burden (because we need to move the localized word "terminal"). `HwndTerminalAutomationPeer` takes this a step further and overrides the class name to be `WPFTermControl`. This allows screen readers to provide special handling for the `WPFTermControl` vs the UWP term control since they will be updating at different speeds. To build the WPF test app, I had to mess with the dependencies a little bit. Really just add the atlas engine and uia renderer to the build steps. The initialization order with `WM_NCCREATE` was changed to match that of Windows Terminal (BaseWindow/IslandWindow). This is safer now. I also removed the `static` window because it was unnecessary. WPF's HwndHost likes to mark the `WM_GETOBJECT` message as handled to force the usage of the WPF automation peer. We now explicitly mark it as not handled and don't return an automation peer. This forces the message to go down to the HwndTerminal where we return terminal's UiaProvider. TermContol (the top-most layer in the UIA tree) would pop up and not do anything. This PR also overrides the automation peer at that layer and marks IsContentElement/IsControlElement=false (the equivalent to AccessibilityView=Raw). This makes the layer only appear in the UIA tree if you are using the raw view (i.e. you know what you're doing and you want to see each individual layer even if you can't directly interact with it). Tested with Narrator/NVDA using WpfTerminalTestNetCore project in our repo. - [X] New output is read out (not just key events, but also other output text) - [X] Local echo does not occur (i.e. pressing 'A' should only read 'A' once, not twice [key event and rendered letter]). - [X] selection events are read out properly - [X] cursor change events are read out properly (tested with text cursor indicator preview in Settings App > Accessibility > Text Cursor) NOTE: test this with Release builds. Debug builds may be too slow and not read out properly Closes #12642 (cherry picked from commit 5608cf1) Service-Card-Id: 86081128 Service-Version: 1.15

Adds UIA events to the WPF control for the following items: - selection changed - text changed (and output) - cursor changed ### Automation Peer Similar to the architecture of the UWP TermControl, we added a `HwndTerminalAutomationPeer` which acts as the `TermControlAutomationPeer` in UWP. However, we don't need a XAML wrapper here, so really we just need it to inherit from `TermControlUiaProvider` (the `ITextProvider` implementation shared across conhost and WT) and `IUiaEventDispatcher` (the event dispatching interface that is responsible for signaling the screen reader that something has changed). ### Removing the local echo As with WT, we need to record key events to remove the local echo. These recorded events are matched up with the output text. Each sequential match is removed in the output text so that it's not read by the screen reader. ### Detecting what to send events for As with WT, a `UiaEngine` was added to the renderer and it is set up when a UIA client is detected. WT would normally stop sending events when focus was lost from the control. We do the same here. ### Automation properties `TermControlUiaProvider` was upgraded to support property values. Such properties include class name and control type. These align with those set in `TermControlAutomationPeer`. Realistically, those should point to these, but that requires a lot more work and a localization burden (because we need to move the localized word "terminal"). `HwndTerminalAutomationPeer` takes this a step further and overrides the class name to be `WPFTermControl`. This allows screen readers to provide special handling for the `WPFTermControl` vs the UWP term control since they will be updating at different speeds. ### Build fixes To build the WPF test app, I had to mess with the dependencies a little bit. Really just add the atlas engine and uia renderer to the build steps. ### HwndTerminal initialization The initialization order with `WM_NCCREATE` was changed to match that of Windows Terminal (BaseWindow/IslandWindow). This is safer now. I also removed the `static` window because it was unnecessary. ### Handling `WM_GETOBJECT` WPF's HwndHost likes to mark the `WM_GETOBJECT` message as handled to force the usage of the WPF automation peer. We now explicitly mark it as not handled and don't return an automation peer. This forces the message to go down to the HwndTerminal where we return terminal's UiaProvider. ### Remove TermControl layer from UIA tree TermContol (the top-most layer in the UIA tree) would pop up and not do anything. This PR also overrides the automation peer at that layer and marks IsContentElement/IsControlElement=false (the equivalent to AccessibilityView=Raw). This makes the layer only appear in the UIA tree if you are using the raw view (i.e. you know what you're doing and you want to see each individual layer even if you can't directly interact with it). ## Validation Steps Performed Tested with Narrator/NVDA using WpfTerminalTestNetCore project in our repo. - [X] New output is read out (not just key events, but also other output text) - [X] Local echo does not occur (i.e. pressing 'A' should only read 'A' once, not twice [key event and rendered letter]). - [X] selection events are read out properly - [X] cursor change events are read out properly (tested with text cursor indicator preview in Settings App > Accessibility > Text Cursor) NOTE: test this with Release builds. Debug builds may be too slow and not read out properly Closes #12642 (cherry picked from commit 5608cf1) Service-Card-Id: 86081129 Service-Version: 1.16

ghost · 2022-10-18T21:50:46Z

🎉Windows Terminal v1.15.2874 has been released which incorporates this pull request.:tada:

Handy links:

ghost · 2022-12-14T20:43:43Z

🎉Windows Terminal Preview v1.16.3463.0 and v1.16.3464.0 has been released which incorporates this pull request.:tada:

Handy links:

ghost added Area-Accessibility Issues related to accessibility Area-WPFControl Things related to the WPF version of the TermControl Issue-Bug It either shouldn't be doing this or needs an investigation. Priority-1 A description (P1) Product-Terminal The new Windows Terminal. labels Sep 28, 2022

carlos-zamora commented Sep 28, 2022

View reviewed changes

carlos-zamora added 3 commits October 3, 2022 13:03

[wpf] Add UIA events

2aa3f3f

enable/disable on focus changed; remove unnecessary bool

bf6f0ff

add UIA properties

9fd73c7

carlos-zamora force-pushed the dev/cazamor/wpf/uia-events branch from 2d8e599 to 9fd73c7 Compare October 3, 2022 20:04

codeofdusk reviewed Oct 4, 2022

View reviewed changes

src/types/TermControlUiaProvider.cpp Show resolved Hide resolved

carlos-zamora added 2 commits October 5, 2022 11:21

static analysis (and one warning)

adf9aa9

lhecker approved these changes Oct 6, 2022

View reviewed changes

DHowett added this to To Cherry Pick in 1.16 Servicing Pipeline via automation Oct 6, 2022

DHowett added this to To Cherry Pick in 1.15 Servicing Pipeline via automation Oct 6, 2022

DHowett reviewed Oct 6, 2022

View reviewed changes

src/cascadia/PublicTerminalCore/HwndTerminal.cpp Outdated Show resolved Hide resolved

DHowett reviewed Oct 6, 2022

View reviewed changes

src/cascadia/PublicTerminalCore/HwndTerminal.cpp Outdated Show resolved Hide resolved

DHowett requested changes Oct 6, 2022

View reviewed changes

ghost added Needs-Author-Feedback The original author of the issue/PR needs to come back and respond to something and removed Needs-Author-Feedback The original author of the issue/PR needs to come back and respond to something labels Oct 6, 2022

fix PR feedback

1e86ac8

DHowett reviewed Oct 6, 2022

View reviewed changes

src/cascadia/PublicTerminalCore/HwndTerminalAutomationPeer.cpp Outdated Show resolved Hide resolved

make bstr nothrow

967e04f

DHowett approved these changes Oct 6, 2022

View reviewed changes

DHowett added the AutoMerge Marked for automatic merge by the bot when requirements are met label Oct 6, 2022

ghost merged commit 5608cf1 into main Oct 6, 2022

ghost deleted the dev/cazamor/wpf/uia-events branch October 6, 2022 23:11

DHowett moved this from To Cherry Pick to Cherry Picked in 1.15 Servicing Pipeline Oct 6, 2022

DHowett moved this from To Cherry Pick to Cherry Picked in 1.16 Servicing Pipeline Oct 6, 2022

carlos-zamora restored the dev/cazamor/wpf/uia-events branch October 7, 2022 18:48

ghost mentioned this pull request Oct 18, 2022

Terminal WPF control doesn't support screen readers #12642

Closed

DHowett moved this from Cherry Picked to Shipped in 1.15 Servicing Pipeline Dec 1, 2022

DHowett moved this from Cherry Picked to Shipped in 1.16 Servicing Pipeline Dec 1, 2022

This pull request was closed.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[wpf] Add UIA events #14097

[wpf] Add UIA events #14097

carlos-zamora commented Sep 28, 2022 •

edited by DHowett

Loading

codeofdusk commented Sep 28, 2022

lhecker Oct 6, 2022

lhecker Oct 6, 2022

carlos-zamora Oct 6, 2022

DHowett left a comment

ghost commented Oct 6, 2022

ghost commented Oct 18, 2022

ghost commented Dec 14, 2022

[wpf] Add UIA events #14097

[wpf] Add UIA events #14097

Conversation

carlos-zamora commented Sep 28, 2022 • edited by DHowett Loading

Automation Peer

Removing the local echo

Detecting what to send events for

Automation properties

Build fixes

HwndTerminal initialization

Handling WM_GETOBJECT

Remove TermControl layer from UIA tree

Validation Steps Performed

codeofdusk commented Sep 28, 2022

lhecker Oct 6, 2022

Choose a reason for hiding this comment

lhecker Oct 6, 2022

Choose a reason for hiding this comment

carlos-zamora Oct 6, 2022

Choose a reason for hiding this comment

DHowett left a comment

Choose a reason for hiding this comment

ghost commented Oct 6, 2022

p.s. you can customize the way I help with merging this pull request, such as holding this pull request until a specific person approves. Simply @mention me (@msftbot) and give me an instruction to get started! Learn more here.

ghost commented Oct 18, 2022

ghost commented Dec 14, 2022

carlos-zamora commented Sep 28, 2022 •

edited by DHowett

Loading

Handling `WM_GETOBJECT`

p.s. you can customize the way I help with merging this pull request, such as holding this pull request until a specific person approves. Simply @mention me (`@msftbot`) and give me an instruction to get started! Learn more here.