New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

Abstract vision framework with included support for focus, navigator object and browse mode caret highlighting #9064

Open
wants to merge 12 commits into
base: master
from

Conversation

Projects
None yet
3 participants
@leonardder
Copy link
Collaborator

leonardder commented Dec 13, 2018

Pr provided on behalf of @BabbageCom
This initial description might be incomplete and is subject to change in the near future.

Link to issue number

Closes #971

Background

For years, NVDA has been an open source screen reader for the blind and partially sighted people who rely on braille and speech output. There are no built-in facilities within NVDA to manipulate the contents of the screen. There are several outstandign issues asking for such features, such as:

  • #7857: Making the screen black (i.e. a screen curtain) while NVDA is active, mainly for privacy reasons
  • #971: Visual highlight of focus, review or browse mode caret location

There is also a long standing desire to have a basic screen magnification facility within NVDA.

This pr intends to lay the base of a vision framework that can be used to implement such functionality in the core of NVDA. It can be considered a fully working prototype, which means that I'm open to any feedback, though rather not something like "do all over again"馃槈

A new vision module

Most magic is supposed to happen within a new vision module. The vision module consists of two distinct concepts:

  • Vision enhancement providers: providers of functionality that change the contents of the screen. A vision enhancement provider can fulfill one or more out of three assistive functions (or roles):

    • Magnifier: to magnify the full screen or a part of it.
    • Highlighter: to highlight important areas of the screen (e.g. the focus, mouse or review position).
    • ColorEnhancer: to change the color presentation of the whole screen or a part of it.

    A vision enhancement provider can implement either one or more of the above assistant functions. For example:

    1. This pr contains a highlighter based on a transparent overlay window on which lines are drawn around the several objects on screen. This NVDA Highlighter has the Highlighter role.
    2. The Screen Curtain add-on, which is a prototype add-on written by me on behalf of @BabbageCom and ported to an add-on by @josephsl, is supposed to be added to NVDA in subsequent pr. It has the role of color enhancer.
    3. Several magnifiers, including ZoomText and the windows magnifier, support multiple roles, i.e. they are able to magnify the screen (magnifier) while converting the coloring to a scheme that is much easier to read for partially sighted people (color enahncer). They can also highlights the focus or mouse location (highlighter). Therefore, if one ever wishes to write an add-on that communicates to an API of a magnifier that also support these additional features, that particular vision enhancement provider can implement two or even three roles.

    In the future, vision enhancement providers can benefit from the functionality to set driver/provider specific options in the gui (#8214).

  • The vision handler, which receives information about focus, foreground, navigator object, review and mouse changes from the core of NVDA. It keeps track of the location of objects on screen and communicates this information to one or more active vision enhancement providers. It keeps track of active enhancement providers and can activate and deactivate them. Therefore, it is very much similar to the braille handler in its approach.

Open questions and issues

  1. The current approach to register a vision enhancement provider, is by calling vision.registerProviderCls. This is called for build-in modules when calling vision.initialize. This behavior is based on the mathPres framework. Alternatively, we could use a similar approach to brailleDisplayDrivers and synthDrivers. The latter has the advantage that vision enhancement providers can be bundled in add-ons without a global pluggin. The more I'm thinking about it, the more I'm preferring this idea. It might be good to make this the first nut to crack in reviewing this pr.

  2. Vision enhancement providers implementing multiple roles can now inherrit from multiple subclasses of VisionEnhancementProviders. Alternatively, we could change the Magnifier, Highlighter and ColorEnhancer classes to be mixin classes. I tend to think that the former is easier to implement for both us and add-on authors.

  3. We need to agree about the naming of new concepts within the vision framework:

    • Roles: magnifier, color enhancer or highlighter
    • Vision enhancement provider: shorten to vision provider, vision driver, ... or leave as is
    • Contexts: focus, caret, etc.
  4. Changes in focus or navigator object are immediately reflected within the framework's internals. Changes in caret, review AND MOUSE positions ARE UPDATED ONCE PER CORE CYCLE. For caret and review, this is by design of the framework, it is similar to braille in this case. For the mouse however, NVDA updates the mouse object and position once per core cycle as well. While this makes sense for blind users who only use the mouse for tracking purposes, magnifiers and highlighters might rely on the mouse position to position the magnified area or the highlight. IN this case, updating once per core cycle is probably too slow.

  5. The refresh timer of the current highlighter implementation is based on a wx timer, which ticks at the main thread I believe. The refresh interval might need some adjustments to work well within all environments.

  6. The same applies to the current colors and the styles of the highlighter. In a later stadium, I intend this to be much more configurable

Proposed review roadmap:

  • Address all questions and issues above
  • Review the main vision module
  • Review the references to the vision module within NVDA's core, i.e. NVDAObjects, appModules, etc.
  • Review the highlighter code and visual look of it

Follow up strategy

@feerrenrut has requested for this framework to be filed into subsequent pull requests. After this pr has been fully reviewed and merged, I intent to file at least two additional pr's:

  1. A graphical interface for the framework
  2. A screen curtain color enhancement provider
@Adriani90

This comment has been minimized.

Copy link
Collaborator

Adriani90 commented Dec 13, 2018

Support from @nishimotz is very welcome I think. As part of the japanese NVDA Team, he is the creator of the add-on focus highlight
https://github.com/nvdajp/focusHighlight
I am sure that the japanese NVDA Team can bring much knowledge in this work as well.

@nishimotz

This comment has been minimized.

Copy link
Contributor

nishimotz commented Dec 16, 2018

I am testing the AppVeyor build (nvda_snapshot_pr9064-16441,7aaee3c0.exe)
and confirmed the default highlighter works if nvda.ini is modified something like this

[vision]
        highlighter = NVDAHighlighter
        [[NVDAHighlighter]]
                highlightFocus = true
                highlightCaret = true
                highlightNavigatorObj = true
except NotImplementedError:
rects = None
if rects:
index = 0 if self.obj.isTextSelectionAnchoredAtStart else -1

This comment has been minimized.

@nishimotz

nishimotz Dec 17, 2018

Contributor

I got errors as follows regarding this line:

ERROR - queueHandler.flushQueue (13:47:02.601):
Error in func _loadBufferDone from eventQueue
Traceback (most recent call last):
  File "queueHandler.pyc", line 53, in flushQueue
  File "virtualBuffers\__init__.pyc", line 464, in _loadBufferDone
  File "browseMode.pyc", line 1152, in event_treeInterceptor_gainFocus
  File "browseMode.pyc", line 1465, in event_gainFocus
  File "vision\__init__.pyc", line 601, in handleGainFocus
  File "vision\NVDAHighlighter.pyc", line 63, in updateContextRect
  File "vision\__init__.pyc", line 270, in updateContextRect
  File "vision\__init__.pyc", line 190, in getContextRect
  File "vision\__init__.pyc", line 205, in _getRectFromTextInfo
NameError: global name 'self' is not defined

leonardder added some commits Dec 17, 2018

vision.VisionHandler.handleUpdate, treat updates to focus objects as 鈥
鈥 gainFocus, since the updated object might have a caret in it and we also want to account for this
@nishimotz

This comment has been minimized.

Copy link
Contributor

nishimotz commented Jan 18, 2019

nvda_snapshot_pr9064-16574,1ca723ff.exe fails initialization as follows:

INFO - core.main (20:56:31.969):
NVDA version pr9064-16574,1ca723ff
INFO - core.main (20:56:31.969):
Using Windows version 10.0.17134 workstation

INFO - core.main (20:56:33.881):
NVDA initialized
ERROR - unhandled exception (20:56:33.881):
Traceback (most recent call last):
  File "wx\core.pyc", line 3240, in <lambda>
  File "vision.pyc", line 545, in setProvider
  File "vision.pyc", line 440, in getProvider
ImportError: No module named NVDAHighlighter

I cannot import this via Python console as well:

>>> import visionEnhancementProviders.NVDAHighlighter
Traceback (most recent call last):
  File "<console>", line 1, in <module>
ImportError: No module named NVDAHighlighter
@leonardder

This comment has been minimized.

Copy link
Collaborator

leonardder commented Jan 18, 2019

Thanks @nishimotz! This should have been fixed in the most recent code.

When looking up scripts, place visionEnhancementProviders higher up i鈥
鈥 the chain. We don't want random global plugins to override vision enhancement provider gesture people might rely on.

Also update some copyright headers
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment