Abstract vision framework with included support for focus, navigator object and browse mode caret highlighting

[LeonarddeR]

Pr provided on behalf of @BabbageCom

Link to issue number

Implements #971, but without a gui

Background

For years, NVDA has been an open source screen reader for the blind and partially sighted people who rely on braille and speech output. There are no built-in facilities within NVDA to manipulate the contents of the screen. There are several outstandign issues asking for such features, such as:

• Give an option to make the computer screen black to nvda #7857: Making the screen black (i.e. a screen curtain) while NVDA is active, mainly for privacy reasons
• Visual highlight of focus, browse or caret location being read. #971: Visual highlight of focus, review or browse mode caret location

There is also a long standing desire to have a basic screen magnification facility within NVDA.

This pr intends to lay the base of a vision framework that can be used to implement such functionality in the core of NVDA. Though there is no GUI in the current pull request, the framework is fully working. The GUI will be added in a follow up pr.

A new vision module

Most magic is supposed to happen within a new vision module. The vision module consists of two distinct concepts:

• Vision enhancement providers: providers of functionality that change the contents of the screen. A vision enhancement provider can be a magnifier, highlighter, or whatever else that changes the screen contents.

  This pr contains a highlighter based on a transparent overlay window on which lines are drawn around the several objects on screen.

  The Screen Curtain add-on, which is a prototype add-on written by me on behalf of @BabbageCom and ported to an add-on by @josephsl, is supposed to be added to NVDA in subsequent pr.

  Vision enhancement providers will also be able to benefit from the functionality to set driver/provider specific options in the gui (Allow setting braille display specific settings in the GUI #8214).

• The vision handler, which receives information about focus, foreground, navigator object, review and mouse changes from the core of NVDA. It keeps track of the location of objects on screen and communicates this information to one or more active vision enhancement providers. This information is communicated to vision enhancement providers using extension points. The handler keeps track of active enhancement providers and can activate and deactivate them. Therefore, it is very much similar to the braille handler in its approach.

Follow up strategy

@feerrenrut has requested for this framework to be filed into subsequent pull requests. After this pr has been fully reviewed and merged, I intent to file at least two additional pr's:

1. A graphical interface for the framework
2. A built-in screen curtain

Known issues

The current highlighter colors might be difficult for some people to see. In the future, I'd like this to be configurable.

Changelog entries:

• Changes for developers
  • A vision framework has been added to NVDA. It allows you to create vision enhancement providers; modules that can change screen contents, optionally based on input from NVDA about object locations.
    • Add-ons can bundle their own providers in a visionEnhancementProviders folder.
    • See the vision and visionEnhancementProviders modules for the implementation of the framework and examples, respectively.

[Adriani90]

Support from @nishimotz is very welcome I think. As part of the japanese NVDA Team, he is the creator of the add-on focus highlight
https://github.com/nvdajp/focusHighlight
I am sure that the japanese NVDA Team can bring much knowledge in this work as well.

[nishimotz]

I am testing the AppVeyor build (nvda_snapshot_pr9064-16441,7aaee3c0.exe)
and confirmed the default highlighter works if nvda.ini is modified something like this

[vision]
        highlighter = NVDAHighlighter
        [[NVDAHighlighter]]
                highlightFocus = true
                highlightCaret = true
                highlightNavigatorObj = true

[nishimotz]

I got errors as follows regarding this line:

ERROR - queueHandler.flushQueue (13:47:02.601):
Error in func _loadBufferDone from eventQueue
Traceback (most recent call last):
  File "queueHandler.pyc", line 53, in flushQueue
  File "virtualBuffers\__init__.pyc", line 464, in _loadBufferDone
  File "browseMode.pyc", line 1152, in event_treeInterceptor_gainFocus
  File "browseMode.pyc", line 1465, in event_gainFocus
  File "vision\__init__.pyc", line 601, in handleGainFocus
  File "vision\NVDAHighlighter.pyc", line 63, in updateContextRect
  File "vision\__init__.pyc", line 270, in updateContextRect
  File "vision\__init__.pyc", line 190, in getContextRect
  File "vision\__init__.pyc", line 205, in _getRectFromTextInfo
NameError: global name 'self' is not defined

[nishimotz]

nvda_snapshot_pr9064-16574,1ca723ff.exe fails initialization as follows:

INFO - core.main (20:56:31.969):
NVDA version pr9064-16574,1ca723ff
INFO - core.main (20:56:31.969):
Using Windows version 10.0.17134 workstation

INFO - core.main (20:56:33.881):
NVDA initialized
ERROR - unhandled exception (20:56:33.881):
Traceback (most recent call last):
  File "wx\core.pyc", line 3240, in <lambda>
  File "vision.pyc", line 545, in setProvider
  File "vision.pyc", line 440, in getProvider
ImportError: No module named NVDAHighlighter

I cannot import this via Python console as well:

>>> import visionEnhancementProviders.NVDAHighlighter
Traceback (most recent call last):
  File "<console>", line 1, in <module>
ImportError: No module named NVDAHighlighter

[LeonarddeR]

Thanks @nishimotz! This should have been fixed in the most recent code.

[feerrenrut]

Except there is a failing unit test, "No module named 'wx'"

[LeonarddeR]

Ah, I'm sorry for that. This all has to do with the linter complaining about module imports have to be at the top of the file.

[LeonarddeR]

@feerrenrut I'm afraid I had to use some noqa in the unit module to get this work. This is because we have to change the path before importing modules, and this is something Flake8 doesn't agree with.

The sourceEnv noqa is intentional, as this module is only imported to manipulate the path.

[feerrenrut]

I'm fine with using noqa to suppress this case, however please include a comment about why and a hint about what the lint error is. It's not much fun for a future developer to have to look these up and understand why it is like this.

[LeonarddeR]

What would be proper syntax for this? I assume something like this is not going to work

Suggested change

	import sourceEnv # noqa: F401
	import sourceEnv # noqa: F401 module imported but unused

[feerrenrut]

Yep something like that works, though it gets messy if you have multiple errors to ignore.

I would do:

# F401: module imported but unused. We need sourceEnv for unit tests because of ?
import sourceEnv  # noqa: F401

[LeonarddeR]

I think I've done something along those lines.

[LeonarddeR]

Mm, I was ignoring linter warning F402, which had to be E402. Interesting enough, Ignoring F402 also ignored E402 unexpectedly, that's why I didn't see this before.

[AppVeyorBot]

PR introduces Flake8 errors 😲

See test results for Failed build of commit 6fbc40b7de

[feerrenrut]

From a unit testing perspective it would be better if we can avoid this. Which part of the vision framework relies on a wx app being available? Can we mock this part instead?

[feerrenrut]

Generally this looks good. But I am a little worried about the dependencies of vision, if possible I would prefer it rely on dependency injection if possible. Can we start by identifying what fails if wx is missing?

[LeonarddeR]

wx is used to make sure that the gui is initialized (CallAfter) before initializing providers. This, because providers might rely on the gui. The highlit"ghter, for example, relies on gui.mainFrame.handle to be the parent of its window. By default, there is no provider active. So it might make sense to check for configured providers before using wx.CallAfter. If we ever decide to write a pseudo provider, this requires another revisit. We can also mock wx.CallAfter, e.g. 'lambda func, *args, **kwargs: func(*args, **kwargs)'. Other code might benefit from this too.