New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow access to Microsoft Word documents via UIAutomation, providing a significant speed improvement, and ability to access documents when in-process injection is impossible. #7849

Merged
merged 25 commits into from Jan 17, 2018

Conversation

Projects
None yet
10 participants
@michaelDCurran
Contributor

michaelDCurran commented Dec 12, 2017

A quick note on testing this pr:
To forecefully use UIAAutomation in Microsoft Word when available, set config.conf['UIA']['useInMSWordWhenAvailable'] to True.
You can confirm that NVDA is using UIA as the role for the document will be document, rather than editableText.
Most importantly though, performance should be a lot snappier.

Link to issue number:

None

Summary of the issue:

Speed improvements

NVDA must fetch a lot of data from Microsoft Word when reading content. This results in many many calls to Microsoft Word's object model.
Although NVDA does many of these calls in-process when possible, recently due to both changes in Microsoft Word, and the growing number of calls needed, performance is getting worse.

Unable to inject

In some situations, such as when running NVDA on Windows 10 S as a store app, it is impossible for NVDA to inject into other processes. Therefore, bulk-fetching data in Microsoft Word by injection is not an option. this results in Microsoft word being completely inaccessible on Windows 10 S for NVDA.

Description of how this pull request fixes the issue:

NVDA already had preliminary support for Microsoft Word via UIAutomation as this was the way NVDA was able to access Windows 10 Mail. However, this support was limited and did not make use of all possible features.
This PR does the following:

Abstraction of table navigation scripts

  • For a while, browseMode has had a pretty nice abstraction of table navigation support. This support has now been moved out into a documentBase.DocumentWithTableNavigation class, which browseMode, and other document implementations can use.
  • Table navigation code was moved out of UIABrowseMode and into a new UIA base class of UIADocumentWithTableNavigation, allowing UIA implementations for browseMode or NVDAObjects to get standard table navigation.
  • The UIAutomation WordDocument NVDAObject now also inherits from this new base class, gaining table navigation support for free.
    In short, table navigation scripts such as control+alt+rightArrow etc, are now available in Microsoft Word when accessed via UIAutomation.

Ensure browseMode works in Microsoft Word with UIAutomation

Some small fixes to UIATextInfo's constructor where needed so that creating a TextInfo from its root UIA NVDAObject would correctly work.

Report more formatting in UIATextInfo

  • Support detection of spelling errors when they are exposed along with other annotations at the same time. Previously we only supported fetching them by themselves.
  • Report the existance of comments.
  • report style name. (E.g. Microsoft Word paragraph style).
  • Report editor revisions including insertions and deletions.
  • Report line spacing.

Further specific Microsoft Word UIA enhansements

  • No longer announce "edit" when entering tables of contents.
  • Report page numbers
  • Ensure to report all spelling errors, not just one for the entire text chunk (if it contained one or more).
  • Report footnotes and endNotes within text.
  • Add a reportCurrentComment script (NVDA+alt+c) which reports the comment (including author and date) similar to Microsoft word support via the object model.

Allow UIA Microsoft Word support in various situations

  • Fallback to UIA support if it is available and injection is impossible
  • Forecefully use UIA if it is available and config.conf['UIA']['useInMSWordWhenAvailable'] is True. This is useful for testing the new UIA support.
  • Allow the UIA support to make use of the existing object model support if available for features such as reporting in format toggle scripts such as turning bold on and off.
    For these changes it was necessary to move some of the older Microsoft Word support into NVDAObjects.IAccessible.winword, and only leave the object model fetching code, and some toggle scripts in NVDAObjects.window.winword.
    Note that in order not to break localization gesture bindings for toggle scripts such as control+b (bold) etc, it was important to keep these toggle scripts in NVDAObjects.window.winword.

Testing performed:

  • Tested with config.conf['UIA']['useInMSWordWhenAvailable'] both true and false, in Microsoft Word 2016, and Windows 10 Mail. this needs much wider testing.

Known issues with pull request:

  • Eventually we may consider turning on UIA support in MS Word by default if available. However, we must be careful to only allow this for complete UIA implementations. I.e. it is possible that Office 2013 exposed a UIA implementation, though it may have been incomplete.
  • Setting / getting of user-defined header rows/columns is not supported. In MS word 2016 and higher it is recommended to actually set the header rows/columns via the Ribbon as that is now supported in Office.
  • Author / date info for track changes in Elements list. This is currently impossible to fetch with MS word's UIA implementation.

Even with these limitations, this is still a far far improvement for situations where injection is unavailable, thus I feel these issues should be addressed in further PRs. this is already a significant patch.

Change log entry:

From the general user's point of view there will not be any change (yet). However, once we start offering NVDA in the Windows store for Windows 10 S, support for Microsoft word will now be possible.
Changes for developers:

  • For document implementations such as NVDAObjects or browseMode that have a textInfo, there is now a new documentBase.documentWithTableNavigation class that can be inherited from to gain standard table navigation scripts. Please refer to this class to see which helper methods must be provided by your implementation for table navigation to work.

michaelDCurran added some commits Sep 21, 2017

Add support for table navigation (control+alt+arrows) to MS word with…
… UIA support, by abstracting existing browseMode table navigation code even further.

Specifically:
Add a new documentBase module which contains:
* TextContainerObject, which is an object that can make a textInfo. It contains a TextInfo class property, as well as makeTextInfo, and selection getter and setter. This is now inherited by NVDAObject and documentTreeInterceptor.
* DocumentWithTableNavigation: a TextContainerObject that contains table navigation scripts, implemented by abstraction table code that used to exist in browseMode. Now BrowseModeDocument inherits from this.
UIABrowseModeDocument's _getTableCellAt was moved into its own UIADocumentWithTableNavigation class, which UIABrowseModeDocument now inherits from, and so dos the UIA WordDocument NVDAObject class, thus now giving UIA word document support table navigation outside browse mode.
UIATextInfo: detect spelling errors when they are a part of multiple …
…annotation types, such as for Microsoft Word.

UIATextInfo: support reporting of comments.

UIATextInfo: support reporting of style name (E.g. Microsoft Word styles)

UIATextInfo: report insertion and deletion revisions (Though Edge and MS Word don't seem to implement this yet).

UIATextInfo: report linespacing.
UIA WordDocumentTextInfo: announce lists correctly and stop announcin…
…g edit in tables of contents.

UIA wordDocumentTextInfo: support page numbers.

UIA Word document support: Always at least split by unit format when fetching formatting. Fetching formatting for the entire text chunk may not be good as some annotationTypes such as spelling return valid results for the total range, and therefore do not give the greatest resolution

UIA WordDocumentTextInfo: remove som old code that detected annoying editable text nodes which no longer works.

UIA word document: report existance of footnotes and endnotes within text.

UIA WordDocument: add reportCurrentComment script
Move some Windows Winword NVDAObject stuff into IAccessible WinWord N…
…VDAObject.

Use UIA with Microsoft word if NVDA connot inject, or NVDA is specifically configured to do so.

Choose WordDocument with IAccessible

Outlook appModule: grab WordDocument code from IAccessible rather than Window

@michaelDCurran michaelDCurran requested a review from feerrenrut Dec 12, 2017

@michaelDCurran michaelDCurran changed the title from Allow access to Microsoft Word documents via UIAutomation, providing a signficicant speed improvement, and ability to access documents when in-process injection is impossible. to Allow access to Microsoft Word documents via UIAutomation, providing a significant speed improvement, and ability to access documents when in-process injection is impossible. Dec 12, 2017

UIA NVDAObject: rowHeaderText and columnHeaderText now filter out the…
… current cell from the headers if inappropriately put there by the UIA implementation.
@feerrenrut

Could you comment on the test plan for this? Knowing what versions / features should be tested would be handy. Then we can ask people to test specific parts.

raise LookupError
UIAGridPattern=None
try:
punk=tableUIAElement.getCurrentPattern(UIAHandler.UIA_GridPatternId)

This comment has been minimized.

@feerrenrut

feerrenrut Dec 12, 2017

Contributor

punk?

@@ -124,7 +124,7 @@ def findOverlayClasses(self,clsList):
from .akelEdit import AkelEdit as newCls
elif windowClassName=="ConsoleWindowClass":
from .winConsole import WinConsole as newCls
elif windowClassName=="_WwG":
elif False: #windowClassName=="_WwG":

This comment has been minimized.

@feerrenrut

feerrenrut Dec 12, 2017

Contributor

Can this block be removed?

if windowClass=="_WwG" and not (config.conf['UIA']['useInMSWordWhenAvailable'] or not appModule.helperLocalBindingHandle):
# Microsoft Word should not use UIA unless we can't inject or the user explicitly chose to use UIA with Microsoft word
return False
return res

This comment has been minimized.

@feerrenrut

feerrenrut Dec 12, 2017

Contributor

The name of this function suggests a boolean return, while this can be implicitly be used as a boolean. I would rather this be changed to return bool(res)

@leonardder

This comment has been minimized.

Collaborator

leonardder commented Dec 12, 2017

@michaelDCurran commented on 12 dec. 2017 03:20 CET:

Change log entry:

From the general user's point of view there will not be any change (yet). However, once we start offering NVDA in the Windows store for Windows 10 S, support for Microsoft word will now be possible.

May be document some changes for developers, as there has been a massive restructure regarding table navigation?

@Brian1Gaff

This comment has been minimized.

Brian1Gaff commented Dec 12, 2017

@derekriemer

This comment has been minimized.

Collaborator

derekriemer commented Dec 12, 2017

  1. Open the following silly example.
  2. Try pressing tab on row 1.
    The expected result occurs.
  3. Press tab on row 6.

Expected:
The next row number is read. Possibly, a new row added notification should be spoken.
bears.docx

Actual:

error executing script: <bound method Dynamic_EditableTextWithoutAutoSelectDetectionWordDocumentUIAWindowNVDAObject.script_tab of <NVDAObjects.Dynamic_EditableTextWithoutAutoSelectDetectionWordDocumentUIAWindowNVDAObject object at 0x08302810>> with gesture u'tab'
Traceback (most recent call last):
  File "scriptHandler.py", line 187, in executeScript
    script(gesture)
  File "NVDAObjects\window\winword.py", line 1268, in script_tab
    info.expand(textInfos.UNIT_CELL)
  File "NVDAObjects\UIA\__init__.py", line 626, in expand
    UIAUnit=UIAHandler.NVDAUnitsToUIAUnits[unit]
KeyError: 'cell'

michaelDCurran added some commits Dec 12, 2017

Winword's script_tab: when we have a collapsed range on a table cell,…
… expand to paragraph rather than cell, as not all implementations can support cell (such as UIA). Paragraph is certainly enough info to speak anyway.
@michaelDCurran

This comment has been minimized.

Contributor

michaelDCurran commented Dec 12, 2017

@derekriemer blank cells should now announce okay with the tab key.

@michaelDCurran

This comment has been minimized.

Contributor

michaelDCurran commented Dec 12, 2017

Notes for testing this pr:

  • Test with many varying word documents, though
    Test document.docx might be useful as it contains headings, formatting changes, spelling errors, comments, revisions, graphics, footnotes and endnotes etc.
  • Testing MS Word 2010 through 2016, with the new UIA code disabled (by default), there should be no user-visible difference at all.
  • With the new UIA code enabled (via the config option):
  • MS word 2010 and 2013: there should be no user-visible change at all.
  • MS Word 2016: UIA should be used. Notice documents have a role of document, and performance is vastly improved. The user should be able to access all functionality they could with the UIA code disabled, even though presentation may be ever so slightly different (E.g. "list" will be reported when entering a list where this does not happen with out UIA code).
  • Test things such as:
  • Headings: reading, quicknav, elements list
  • links: reading, quicknav, elements list
  • tables: reading, table navigation commands, quick nav
  • announcing existance of revisions, spelling errors,
  • announcing existance of comments, reporting current comment
  • reading with sayAll
  • announcement of page numbers
  • reporting of formatting, and auto announcement of format changes (with options turned on in Document formatting settings)
  • announcement of styles
  • Reporting of toggle formatting commands (e.g. bold control+b), italic, underline, line spacing, outline level etc.

Reiterating again that the main purpose of this code for now is to provide something when injection is not available (such as in windows 10 S).
Therefore, crashes, errors, things not working at all, are the most important. Small presentation fixes can be addressed later.

michaelDCurran added some commits Dec 12, 2017

Allow editableText's newLine script to announce the entire new line i…
…f announceEntireNewLine class viarable is true. This is false by default, but UIA WordDocument sets this to true for better list bullet reading.
UIA NVDAObject: add the EditableTextWithOoutAutoSelectDetection mixin…
… class at the very end before the base UIA class, so that subclasses of UIA can override methods and properties of EditableTextWithoutAutoSelectDetection (such as for controlling of announcement of new line text).
UIA WordDocument: for now suppress announcement of entering and exiti…
…ng lists. this ensures compatibility with the older MS Word implementation, and stops confusion when pressing enter for new bullets where NVDA was announcing exiting and then entering a new list.
from displayModel import EditableTextDisplayModelTextInfo
from NVDAObjects.window import DisplayModelEditableText
from ..behaviors import EditableTextWithoutAutoSelectDetection
from NVDAObjects.window.winword import *

This comment has been minimized.

@derekriemer

derekriemer Dec 21, 2017

Collaborator

Would a comment here for explanation of what this file is for be worth putting?

elif obj.UIAElement.cachedControlType==UIAHandler.UIA_CustomControlTypeId and obj.name:
# Include foot note and endnote identifiers
field['content']=obj.name
#field['alwaysReportName']=True

This comment has been minimized.

@derekriemer

derekriemer Dec 21, 2017

Collaborator

Is there a reason this is commented?

# To stay compatible with the older MS Word implementation, don't expose lists in word documents as actual lists. This suppresses announcement of entering and exiting them.
# Note that bullets and numbering are still announced of course.
# Eventually we'll want to stop suppressing this, but for now this is more confusing than good (as in many cases announcing of new bullets when pressing enter causes exit and then enter to be spoken).
field['role']=controlTypes.ROLE_EDITABLETEXT
if obj.role==controlTypes.ROLE_GRAPHIC:
# Label graphics with a description before name as name seems to be auto-generated (E.g. "rectangle")
field['value']=field.pop('description',None) or obj.description or field.pop('name',None) or obj.name

This comment has been minimized.

@derekriemer

derekriemer Dec 21, 2017

Collaborator

Will this break Microsoft's AI generated image descriptions possibly?

try:
UIAElementArray=val.QueryInterface(UIAHandler.IUIAutomationElementArray)
except COMError:
print "not an array"

This comment has been minimized.

@derekriemer

derekriemer Dec 21, 2017

Collaborator

Debugging code.

@@ -139,7 +140,7 @@ def clearDynamicClassCache(cls):
"""
cls._dynamicClassCache.clear()
class NVDAObject(baseObject.ScriptableObject):
class NVDAObject(documentBase.TextContainerObject,baseObject.ScriptableObject):
"""NVDA's representation of a single control/widget.
Every widget, regardless of how it is exposed by an application or the operating system, is represented by a single NVDAObject instance.
This allows NVDA to work with all widgets in a uniform way.

This comment has been minimized.

@derekriemer

derekriemer Dec 21, 2017

Collaborator

Does that change need a docs update?

@@ -312,7 +312,13 @@ def _isUIAWindowHelper(self,hwnd):
if appModule and appModule.isBadUIAWindow(hwnd):
return False
# Ask the window if it supports UIA natively
return windll.UIAutomationCore.UiaHasServerSideProvider(hwnd)
res=windll.UIAutomationCore.UiaHasServerSideProvider(hwnd)

This comment has been minimized.

@derekriemer

derekriemer Dec 21, 2017

Collaborator

Just curious, why is this being changed for this project?

return windll.UIAutomationCore.UiaHasServerSideProvider(hwnd)
res=windll.UIAutomationCore.UiaHasServerSideProvider(hwnd)
if res:
# the window does support UIA natively, but

This comment has been minimized.

@derekriemer

derekriemer Dec 21, 2017

Collaborator

That comment isn't a sentence.

import ui
import controlTypes
class TextContainerObject(AutoPropertyObject):

This comment has been minimized.

@derekriemer

derekriemer Dec 21, 2017

Collaborator

There is no docstring here.

info.updateSelection()
class DocumentWithTableNavigation(TextContainerObject,ScriptableObject):

This comment has been minimized.

@derekriemer

derekriemer Dec 21, 2017

Collaborator

Same here.

michaelDCurran added some commits Jan 3, 2018

UIA MS Word support: allow backspace to work again. Seems like range …
…comparison is now broken in MS Word's UIA implementation as well.

michaelDCurran added a commit that referenced this pull request Jan 3, 2018

@josephsl

This comment has been minimized.

Collaborator

josephsl commented Jan 3, 2018

Hi,

For @michaelDCurran: please keep an eye on #7456, as your latest commit fixes it for now. Thanks.

@josephsl

This comment has been minimized.

Collaborator

josephsl commented Jan 4, 2018

Hi,

Reported by a user (paraphrased): When opening Start menu in Windows 7, an error is recorded.

STR:

  1. From Windows 7, install latest next snapshot.
  2. Open Start menu.

Expected: no errors.
Actual:

ERROR - eventHandler.executeEvent (21:12:00.062):
error executing event: gainFocus on <NVDAObjects.Dynamic_EditableTextWithoutAutoSelectDetectionUIA object at 0x05DFBEF0> with extra args of {}
Traceback (most recent call last):
File "eventHandler.pyc", line 152, in executeEvent
File "eventHandler.pyc", line 92, in init
File "eventHandler.pyc", line 100, in next
File "C:\Users\Angelo\AppData\Roaming\nvda\addons\remote\globalPlugins\remoteClient_init_.py", line 410, in event_gainFocus
File "eventHandler.pyc", line 100, in next
File "appModules\explorer.pyc", line 280, in event_gainFocus
File "eventHandler.pyc", line 100, in next
File "NVDAObjects_init_.pyc", line 951, in event_gainFocus
File "NVDAObjects_init_.pyc", line 839, in reportFocus
File "speech.pyc", line 396, in speakObject
File "speech.pyc", line 752, in speakTextInfo
File "NVDAObjects\UIA_init_.pyc", line 621, in getTextWithFields
File "NVDAObjects\UIA_init_.pyc", line 611, in getTextWithFieldsForUIARange
File "NVDAObjects\UIA_init
.pyc", line 446, in getTextWithFields_text
File "NVDAObjects\UIA_init
.pyc", line 241, in _getFormatFieldAtRange
UnboundLocalError: local variable 'fetchAnnotationTypes' referenced before assignment

Thanks.

@zstanecic

This comment has been minimized.

Contributor

zstanecic commented Jan 4, 2018

@josephsl

This comment has been minimized.

Collaborator

josephsl commented Jan 4, 2018

@zstanecic

This comment has been minimized.

Contributor

zstanecic commented Jan 5, 2018

@Brian1Gaff

This comment has been minimized.

Brian1Gaff commented Jan 5, 2018

@zstanecic

This comment has been minimized.

Contributor

zstanecic commented Jan 5, 2018

@ruifontes

This comment has been minimized.

ruifontes commented Jan 6, 2018

@zstanecic

This comment has been minimized.

Contributor

zstanecic commented Jan 6, 2018

@leonardder

This comment has been minimized.

Collaborator

leonardder commented Jan 6, 2018

I found another browse mode bug regarding finding:

  1. Enter the following lines of text:

    Hello!
    How are you?
    
  2. Go to the top of the document

  3. Enable browse mode

  4. Search for how

It seems that the cursor does move one cursor position to the right, but it does not move to the actual result.

@fernando-jose-silva

This comment has been minimized.

fernando-jose-silva commented Jan 8, 2018

Sorry for opening a separate call to this discussion.
As I reported in the call I opened 7903, I would just like to report that I can not read characters, words or lines when editing a cell in excel 2013 on windows 7.
I saw that as far as the menu to start this had already been approached in this discussion.
Thank you.

UIATextInfo: fix an AttributeError on Windows 7 by defining the fetch…
…AnnotationTypes variable before any point where it can be used. Specifically: If the IUIAutomationTextRange3 interface was missing, fetchAnnotationTypes was never set. It should be set to False by default.

michaelDCurran added a commit that referenced this pull request Jan 11, 2018

@michaelDCurran

This comment has been minimized.

Contributor

michaelDCurran commented Jan 11, 2018

The issue with find in browse mode is clearly a bug in MS Word's UI automation implementation. If it finds text, it returns the original text range, rather than the position of the found string. There is no way to work around this in NVDA.

@michaelDCurran

This comment has been minimized.

Contributor

michaelDCurran commented Jan 11, 2018

Re grammar errors crash with Elements list: a test word document file would be great.

@michaelDCurran

This comment has been minimized.

Contributor

michaelDCurran commented Jan 11, 2018

and the Windows 7 attributeError in the start menu has been fixed in the last commit to this branch.

@zstanecic

This comment has been minimized.

Contributor

zstanecic commented Jan 11, 2018

@zstanecic

This comment has been minimized.

Contributor

zstanecic commented Jan 11, 2018

@Brian1Gaff

This comment has been minimized.

Brian1Gaff commented Jan 12, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment