Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Preliminary support for UI Automation in Windows Console #9614

Merged
merged 26 commits into from May 27, 2019

Conversation

@codeofdusk
Copy link
Collaborator

commented May 23, 2019

Link to issue number:

This PR shouldn't yet close any issues, but it is related to #513, #673, #1682, #6291, and #7497.

Summary of the issue:

NVDA provides support for the Windows command console used by Command Prompt, PowerShell, and the Windows Subsystem for Linux. It allows users to read the console's contents using review cursor commands, and automatically reports console output as it arrives. While functional, this support has some shortcomings, leading to decreased NVDA performance and stability and, by extension, user productivity.

Description of how this pull request fixes the issue:

This PR re-implements NVDA's Windows Console support using Microsoft's UI Automation API (UIA). At the moment, typed characters and words are read to the user as well as incoming text. Review of console output is functional on Windows 10 1803 and 1809. The new console support is implemented in the NVDAObjects.UIA.winConsoleUIA.winConsoleUIA object, and can be enabled with a new option in NVDA's advanced preferences.

Since the UIA console interface was first implemented in Windows 10 version 1709, it is safe to use the keyboardHandler module to receive typed characters in UIA consoles. The console sometimes sends duplicate WM_CHAR messages for typed characters, so the in-process approach makes "speak typed words" unreliable.

On windows 10 version 1903 and later, the UIA caret position is off-by-one. To compensate, consoleUIATextInfo moves new UIA text ranges around the caret one character to the right.

A mapping from UIA's event for text changes to NVDA's textChange event was present, but commented out, in _UIAHandler. This mapping has been uncommented to allow automatic readout of console text to be implemented.

Automatic text readout uses the NVDAObjects.behaviors.LiveText functionality. Since UIA adds thousands of empty lines to console output, it was necessary to re-implement _getTextLines to filter the output.

UIA fires textChange whenever the console's text changes, even during user input. While LiveText already filters out many typed characters, some duplicates (particularly the beginning of new words) were being reported. To compensate, winConsoleUIA does not report any new text while the user is typing. This is not a regression of NVDA's functionality, as speech is already cancelled during user input, even when "speak typed characters" and "speak typed words" are disabled.

Testing performed:

Tested the new support on Windows 10 versions 1803, 1903, and 2003. In particular, tested reporting of typed characters and words (when the corresponding settings were enabled), reporting of new text, text editing and selection, and object review.

Known issues with pull request:

  • In object review, commands for moving to and reporting words move to or report lines instead.
  • A large number (thousands in many cases) of empty lines are appended to the console's object review.
  • Reporting of text selection is incorrect on Windows 10 version 1809 and earlier.
  • Sometimes, automatic readout begins reading the terminal window from the top instead of reporting only new text (I think this is related to printing empty lines or clearing the screen, it can be reproduced when closing the nano text editor). Maybe fixed in 9d43eb0.
  • When "speak typed characters" or "speak typed words" is enabled, typed text that is not displayed onscreen (such as passwords) is still reported.
  • After all characters of the input line have been deleted, further presses of backspace read the last character of the output line.
  • When reviewing to the last line of the console, the review cursor may become stuck. When this happens, it is impossible to review text in the console until moving focus away from the window and back again.
  • Sometimes, attempting to interact with the console too quickly after focusing the window can make NVDA unable to get any caret or review information or receive new text events. To work around this, move focus away from the window and back again.
  • In console applications that update the screen while the user is typing (i.e. before winConsoleUIA._isTyping is cleared), reporting of new text and speak typed words may not function correctly.
  • If the advanced setting "force UI Automation in the Windows Console" is enabled on a system without a UIA console (such as Windows versions below Windows 10 1709 or systems with the "use legacy console" option selected), the console is completely inaccessible.
  • If "speak typed words" is enabled, the last typed word is announced when pressing enter.
  • When pressing an interrupt character (such as control+c), the _isTyping flag is not cleared on the console, so new text is not announced immediately.

Change log entry:

== New features ==

  • Added an experimental option to the Advanced Settings panel that allows you to try out a new, work-in-progress rewrite of NVDA's Windows Console support using the Microsoft UI Automation API. (#9614)

== Changes for Developers ==

  • Added a new isAtLeastWin10 function to the winVersion module which returns whether or not this copy of NVDA is running on at least the supplied release version (such as 1809 or 1903) of Windows 10. (#9614)

codeofdusk added some commits May 14, 2019

Add a feature flag for UIA console support.
At the moment, it merely disables NVDA's existing console support, falling back to what UIA support is available in NVDA. The option is currently called `ConsoleUIA`, but we'll need to change this once autodetection is implemented (as there will be three options at that point).

Typed characters are now spoken in UIA consoles, but characters are doubled when typing quickly, leading to incorrect typed word reporting.
Disable in-process injection for speak typed characters for now.
This fixes the doubled characters bug, making speaked typed words function as intended, but it may have other implications.
Receive textChanged events from UIA and add consoleUIA and consoleUIA…
…TextInfo classes to implement automatic readout and work around a caret movement bug in Windows 10 version 1903.

The consoleUIA class inherits from NVDAObjects.behaviors.Terminal, which serves as the foundation for automatic readout support. At the moment, it is likely that such support will either be not functional at all or report extraneous text from user input.
Introduce basic automatic readout support, and rename ConsoleUIATextI…
…nfo->consoleUIATextInfo for consistency.

See the source code comments in this commit for some steps still incomplete.
Filter out extraneous characters by disabling live text reporting whi…
…le a user is typing.

This does not regress NVDA functionality, since we already cancel speech when the user types.
Style, eliminate magic number, halve typing timeout, update label in …
…GUI to inform the user that restarting NVDA is required to switch console implementations.
Re-implement consoleUIA._getTextLines using UIATextPattern.GetVisible…
…Ranges. This may improve performance when large amounts of text are written to the console, but possibly at the cost of less reliable autoread (particularly when the screen is refreshed or scrolled).

@feerrenrut feerrenrut requested review from michaelDCurran and feerrenrut May 24, 2019

@feerrenrut
Copy link
Contributor

left a comment

A few small things, overall it looks good! Could you also update the user guide to mention the new setting too.

source/NVDAObjects/UIA/ConsoleUIA.py Outdated Show resolved Hide resolved
source/gui/settingsDialogs.py Outdated Show resolved Hide resolved
source/NVDAObjects/UIA/ConsoleUIA.py Outdated Show resolved Hide resolved
Review actions, style, update user guide.
The user guide will need to be updated once auto-detection is introduced (and therefore the NVDA preference is changed).
@leonardder
Copy link
Collaborator

left a comment

This really looks promising.

source/NVDAObjects/UIA/ConsoleUIA.py Outdated Show resolved Hide resolved
source/NVDAObjects/UIA/ConsoleUIA.py Outdated Show resolved Hide resolved
source/NVDAObjects/UIA/ConsoleUIA.py Outdated Show resolved Hide resolved
source/NVDAObjects/UIA/ConsoleUIA.py Outdated Show resolved Hide resolved
source/NVDAObjects/UIA/ConsoleUIA.py Outdated Show resolved Hide resolved
source/_UIAHandler.py Outdated Show resolved Hide resolved
source/config/configSpec.py Outdated Show resolved Hide resolved
source/gui/settingsDialogs.py Outdated Show resolved Hide resolved
source/keyboardHandler.py Outdated Show resolved Hide resolved
source/keyboardHandler.py Outdated Show resolved Hide resolved
@codeofdusk

This comment has been minimized.

Copy link
Collaborator Author

commented May 25, 2019

@feerrenrut @leonardder I think I've addressed all review items. Could you please have another look?

source/NVDAObjects/UIA/winConsoleUIA.py Outdated Show resolved Hide resolved
source/_UIAHandler.py Outdated Show resolved Hide resolved
source/config/configSpec.py Outdated Show resolved Hide resolved

codeofdusk added some commits May 25, 2019

Fix document review on Windows 10 1903.
This fix adds a new textInfo instance variable, _expandCollapseBeforeReview, which stops review from expanding and immediately collapsing the textInfo during review. For some reason, this was causing object review to malfunction in this instance.
source/_UIAHandler.py Outdated Show resolved Hide resolved
source/_UIAHandler.py Outdated Show resolved Hide resolved
source/winConsoleHandler.py Outdated Show resolved Hide resolved
source/NVDAObjects/UIA/winConsoleUIA.py Outdated Show resolved Hide resolved

codeofdusk added some commits May 27, 2019

Check bad UIA window class name at runtime.
This allows for UIA console support to be dynamically enabled/disabled.

Note: _UIAHandler.badUIAWindowClassNames has been removed. To check if a UIA windowClassName is bad, use the new _UIAHandler.UIAHandler._isBadUIAWindowClassName method.
Add a new winVersion.isAtLeastWin10 function for checking win10 build…
…s with friendly version numbers, and only register for UIA textChange on Windows 10.
@codeofdusk

This comment has been minimized.

Copy link
Collaborator Author

commented May 27, 2019

Could you please have another look @michaelDCurran ?

source/_UIAHandler.py Show resolved Hide resolved
source/winVersion.py Show resolved Hide resolved
source/winVersion.py Outdated Show resolved Hide resolved
@leonardder
Copy link
Collaborator

left a comment

Arrg, really frustrating that GitHub calls every way to comment on an ongoing conversation for an inline comment that is outdated. I'm taking the discussion about the config spec to the main threat here.

Ah, I probably should've clarified a few things:

  • Windows 10 has had UIA console support since version 1709, but it was incomplete in early builds (see #7497). The final PR will set a minimum good Windows version for UIA, which likely won't be 1709 (even though 1709 has limited support). That is to say, we won't turn on UIA console support automatically until a Windows version where we know it works completely, but we should give users the option of forcibly enabling it if our default is too aggressive.

I agree that this makes sense.

  • To accomplish this, I proposed the provisional setting during development (and then adding the auto-detection and new setting once it's final), but maybe there's a better way?

As noted, a provisional setting requires a config spec and scheme update as soon as it gets changed. Especially if you already know what the final setting is going to look like, I'd prefer just adding the setting to the spec as it is supposed to be in the final pr. You could map UIA to True and legacy to False and just make auto act like False/legacy for now.

source/_UIAHandler.py Show resolved Hide resolved
try:
return winVersion.build >= win10VersionsToBuilds[version]
except KeyError:
log.warning("Unknown Windows 10 version {}".format(version))

This comment has been minimized.

Copy link
@leonardder

leonardder May 27, 2019

Collaborator

I'd rather see this raise an exception, but good to know opinions from others as well here.

return winVersion.build >= win10VersionsToBuilds[version]
except KeyError:
log.warning("Unknown Windows 10 version {}".format(version))
return False

This comment has been minimized.

Copy link
@leonardder

leonardder May 27, 2019

Collaborator

Please add an empty line below this line.

@codeofdusk

This comment has been minimized.

Copy link
Collaborator Author

commented Jun 5, 2019

@beqabeqa473 Fixed in #9647.

feerrenrut added a commit that referenced this pull request Jul 31, 2019

Ignore _expandCollapseBeforeReview in script_reviewStartOfLine. (PR #…
…9944)

This fixes an issue in UIA consoles, invoking the "review start of line" script (`shift+numpad 1` by default) does not move the review cursor.

Expanding and collapsing the textInfo is essential for the functionality of this script (to move the textInfo to the start of the line), we should ignore the _expandCollapseBeforeReview flag.

The _expandCollapseBeforeReview flag was added in #9614. It is True everywhere except in UIA consoles.

feerrenrut added a commit that referenced this pull request Aug 1, 2019

Improve keyboard support for some terminal programs (PR #9915)
Previously:
NVDA failed to announce typed characters and/or words in Mintty, and spells output close to the caret in 
legacy Windows consoles.

This commit factors out much of the code for handling typed characters in UIA consoles into a new
`NVDAObjects.behaviors.TerminalWithoutTypedCharDetection class`. The class is now used in 
Mintty (PuTTY, Git Bash) and legacy Windows consoles on Windows 10 version 1607 and later.

In legacy Windows consoles, the old keyboard handling code is disabled when the class is in use, and the
new support can be disabled in the advanced preferences in case it is incompatible with some programs 
or if suppression of passwords is critical.

Since legacy Windows consoles fire textChange rather slowly, this commit prefers faster responsiveness at 
the cost of offscreen characters (such as passwords) always being reported. Users may disable "speak typed 
characters" and/or "speak typed words" (using the existing scripts) when entering passwords to suppress 
this output.

On Windows 10 version 1607 with the new keyboard support enabled, spurious characters are reported 
when the dead key (if available) is pressed.

Fixes #513 
Fixes #1348
Related to #9614

feerrenrut added a commit that referenced this pull request Aug 1, 2019

UI Automation in Windows Console: improve reliability of visible rang…
…e checks (PR #9957)

Builds on #9614
Supersedes #9735 and #9899
Closes #9891

Previously, after the console window was maximized (or the review cursor is otherwise placed outside the visible text), text review is no longer functional.
The review top line and review bottom line scripts do not function as intended.

This commit changes:
- The isOffscreen property has been implemented as UIAUtils.isTextRangeOffscreen.
- When checking if the text range is out of bounds, we now also check that oldRange is in bounds before stopping movement.
- Re-implemented POSITION_FIRST and POSITION_LAST in terms of visible ranges.

feerrenrut added a commit that referenced this pull request Aug 2, 2019

UI Automation in Windows Console: Make UIA the default for Windows 10…
… 1803 and later (PR #9771)

NVDA's UIA console support significantly improves performance and stability, so we should enable it by default.

While UIA in consoles was added in Windows 10 1709, the initial implementation left much to be desired, particularly concerning caret movement and expansion to character.

A new `UIAUtils.shouldUseUIAConsole` function has been added which determines whether to use UIA in consoles:
* If the user has explicitly chosen to use UIA or legacy console support, the function returns `True` or `False` respectively.
* Otherwise, if the user is running at least Windows 10 1803, the function returns `True`, or `False` if not.

The "use UI Automation in the Windows Console when available" GUI setting has been changed to expose all options in the config spec (i.e. to automatically detect UIA/legacy or override this autodetection).

builds on #9614
closes #673
closes #6291

feerrenrut added a commit that referenced this pull request Aug 12, 2019

UIA in Win Console: fix setEndPoint/compareEndPoints (PR #10057)
Closes #10035
Related to #9614
Identical to #10043

Works around a UIA bug on Windows 10 1803 and later that means we can
not trust the "end" endpoint of a collapsed (empty) text range for comparisons. 
The console incorrectly reports the "end" as being past the "start" endpoint.

This stops braille being able to properly track the system caret in Windows Console.

Instead use getText(1) on the textRange to attempt to fetch 1 character.
getText returns an empty string for a collapsed range. By definition, the "start" and
"end" endpoints for a collapsed range are equivalent, thus read from the "start"
endpoint of a collapsed range instead of the "end" endpoint.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
7 participants
You can’t perform that action at this time.