Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

UI Automation in Windows Console and Windows Terminal: block SV2M2 notification events for now and preserve accessibility after UIA class name change #13261

Merged
merged 4 commits into from
Feb 16, 2022

Conversation

codeofdusk
Copy link
Contributor

@codeofdusk codeofdusk commented Jan 20, 2022

Link to issue number:

Related to microsoft/terminal#12358 among others (to be released).

Summary of the issue:

In upcoming Windows Terminal and Windows Console, UIA notification events will be sent when new text is inserted (i.e. for text output) to improve Narrator support. This will result in double-reporting of all terminal output: once from LiveText and once from UIA notifications (but without appropriate filtering for typed characters/passwords etc).

As part of the implementation of notifications, I asked Microsoft to change the UIA class name to allow new terminal support (including notifications) in a follow-up NVDA PR, since notifications will require a departure from the terminal strategy used by NVDA in the past. Changing the class name will break NVDA's current ability to identify the terminal and implement accessibility.

Description of how this pull request fixes the issue:

  • Suppress any received UIA notification events in UIA console and terminal for now, but log them for development.
  • Support the new UIA class name used by terminals that send notifications.

Ideally, NVDA would use these events in place of LiveText. If we could get away without registering for TextChange at all, this could be an extreme stability improvement in the terminal as #11002 would be completely circumvented. Also, having new text pushed to us (instead of having to diff) should improve performance considerably in high-volume scenarios.

Testing strategy:

Tested that support works as expected with the new terminal.

Known issues with pull request:

It is imperative that this PR be included in 2022.1, as upcoming builds of Windows Terminal will have severely degraded accessibility without it.

Change log entries:

None (workaround for bug in unreleased code)

Code Review Checklist:

  • Pull Request description:
    • description is up to date
    • change log entries
  • Testing:
    • Unit tests
    • System (end to end) tests
    • Manual testing
  • API is compatible with existing add-ons.
  • Documentation:
    • User Documentation
    • Developer / Technical Documentation
    • Context sensitive help for GUI changes
  • UX of all users considered:
    • Speech
    • Braille
    • Low Vision
    • Different web browsers
    • Localization in other languages / culture than English

@josephsl
Copy link
Collaborator

josephsl commented Jan 20, 2022 via email

@codeofdusk
Copy link
Contributor Author

codeofdusk commented Jan 20, 2022

Not yet, staying ahead of upcoming changes, though this PR doesn't break existing functionality and should be merged to ensure stability when the changes land.

CC @carlos-zamora, @DHowett.

@codeofdusk codeofdusk marked this pull request as ready for review January 20, 2022 03:05
@codeofdusk codeofdusk requested a review from a team as a code owner January 20, 2022 03:05
@codeofdusk
Copy link
Contributor Author

CC @seanbudd.

@seanbudd seanbudd added this to the 2022.1 milestone Jan 27, 2022
@seanbudd
Copy link
Member

Is there a way to confirm that the console/terminal changes are guaranteed to be implemented?
Is there a way to test this PR with the proposed console/terminal changes?

@codeofdusk
Copy link
Contributor Author

Is there a way to confirm that the console/terminal changes are guaranteed to be implemented?

"guarantee" is hard, but wt will become default, Narrator support is blocking, and notifications is the current path to enable this, so it is all but certain at this point.

Is there a way to test this PR with the proposed console/terminal changes?

I don't yet have access to a build with notifications available, but I've tested this event handler in other contexts that emit UIA notifications (Word with UIA) and it works as intended. Merging this PR is strictly better than the current behaviour and results in no change in situations where UIA notifications are not emitted.

@seanbudd
Copy link
Member

I think it might be worth holding off on merging this until a Windows Insider build comes that it can be tested with.
Considering this is a small, low risk change, it can be merged into alpha late in the release cycle (eg just before we release beta).

@codeofdusk
Copy link
Contributor Author

codeofdusk commented Jan 28, 2022

I'll work on obtaining a Windows Terminal build with which I can test this. Thanks for adding to the 2022.1 release.

@josephsl
Copy link
Collaborator

josephsl commented Jan 28, 2022 via email

@carlos-zamora
Copy link

Hi! I'm over from the Console Engineering team at Microsoft. I just wanted to follow up on some things from this thread.

wt will become default

Yes! We're looking to do this this year! Here's our blog post announcement: https://devblogs.microsoft.com/commandline/windows-terminal-as-your-default-command-line-experience/

Is there a way to confirm that the console/terminal changes are guaranteed to be implemented? Is there a way to test this PR with the proposed console/terminal changes?

We're making progress. I'll be working on creating a prototype this month and sharing that with Bill, NVDA, and JAWS (I'll be meeting with them next week to provide support). We're hoping that this'll be the fastest way to get Narrator on board since this change will have a pretty severe impact on Narrator users.

Like Bill, I hope to take a look at what other text providers (like Word) are doing. So if this works with Word already, that's very promising.

CC @DHowett

@codeofdusk
Copy link
Contributor Author

I can confirm that, with the build in microsoft/terminal#12358, duplicate (buggy) output is present without this PR and absent (correct behaviour) with this PR.

@codeofdusk codeofdusk force-pushed the cmduia-block-notifications branch 2 times, most recently from 8812a1d to 08ffe3f Compare February 4, 2022 02:05
@codeofdusk codeofdusk changed the title UI Automation in Windows Console and Windows Terminal: block SV2M2 notification events for now UI Automation in Windows Console and Windows Terminal: block SV2M2 notification events for now and preserve accessibility after UIA class name change Feb 11, 2022
@feerrenrut feerrenrut removed this from the 2022.1 milestone Feb 14, 2022
@codeofdusk
Copy link
Contributor Author

Now that this is known to work with an actual implementation, can this PR please be merged?

Copy link
Member

@seanbudd seanbudd left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @codeofdusk. It's great that we can prepare for this change early, so that users won't have to update NVDA from 2022.1 to fix this.

Do we have an estimate at what time or which Windows releases this will be incorporated into?
I think it would be helpful to add a changelog item, as users on older versions of NVDA will want to search for which NVDA version fixes this bug.

@codeofdusk
Copy link
Contributor Author

codeofdusk commented Feb 16, 2022

It will arrive in the Sun Valley 2 console and a Windows Store update. Can't speak to anything beyond that sadly. (there is precedent for making fixes to prepare for unreleased software without changelog items, see #7497).

@seanbudd
Copy link
Member

@codeofdusk Thanks, perhaps for users "Windows 11" is specific enough. It is helpful to add bug fixes to the changelog where possible.
Would it be accurate to add the following:

- Prevents a bug causing double-reporting when using Windows 11 Console and Terminal. (#13261)

@codeofdusk
Copy link
Contributor Author

codeofdusk commented Feb 16, 2022

I'd remove "11" as I think they'll backport the terminal code to vibranium at least. So just "Windows Console and Terminal".

@josephsl
Copy link
Collaborator

josephsl commented Feb 16, 2022 via email

@seanbudd
Copy link
Member

Hi, for reference, Vibranium = Windows 10, so a compromise could be to say “recent Windows 10 and 11” (I understand that the what’s new entry was committed, but hope this could be handy in the future). Thanks.

I think the problem here is that "recent" is relative and won't be helpful in the future. With the uncertainty surrounding which versions this will be included in, as @codeofdusk suggested potentially earlier builds to Vibranium, I opted to just go with "Windows".

@josephsl
Copy link
Collaborator

josephsl commented Feb 16, 2022 via email

@seanbudd seanbudd merged commit 24e605b into nvaccess:master Feb 16, 2022
@nvaccessAuto nvaccessAuto added this to the 2022.1 milestone Feb 16, 2022
@codeofdusk
Copy link
Contributor Author

Please note that, due to pushback from JAWS, the TermCTRL2 class name is currently not used. However NVDA should maintain support for it in case this changes in the future.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

6 participants