Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MS Word with UIA: use custom extensions to support line number, layout column number and section number #13283

Merged
merged 30 commits into from
Feb 16, 2022

Conversation

michaelDCurran
Copy link
Member

Link to issue number:

None.

Summary of the issue:

When supporting Microsoft word via UI automation, up to now there was no way to report line numbers, layout column numbers or section numbers. This was because these concepts do not exist in the UI Automation standard.
However, Microsoft recently added custom patterns to UI automation, as a part of its Custom Extensions work, which included custom properties and custom annotations.
Microsoft word has now added its own custom pattern which allows fetching information from a text range, such as line number, layout column number and section number.

Description of how this pull request fixes the issue:

Using the microsoft/microsoft-ui-uiautomation Remote Operations library, fetch line number, layout column number and section number in Microsoft word.
Note that the Remote Operations library is only supported in Windows 11 and above, and the custom pattern in Microsoft Word is only available as of build 16.0.14725.

A part from adding support for these specific attributes, this PR also acts as a first commit to add integrations with the UI Automation Remote Operations library. Now that we are building and including this library, there is much more code that can eventually be remoted for a massive speedup.

Testing strategy:

  • Opened a document in Microsoft word
  • Turned on reporting of line numbers in NVDA's document formatting settings
  • Moved up and down the document with the arrow keys, ensuring that NVDA spoke the current line number before the content of the line.
  • Ensured report pages was turned on in NVDA's document formatting settings.
  • Set up the document in 2-column layout, and inserted a column break in the content.
  • used the arrow keys to move between the two columns, ensuring that the column number is announced as it changes.

Known issues with pull request:

None known.

Change log entries:

New features
Changes
Bug fixes
For Developers

Code Review Checklist:

  • Pull Request description:
    • description is up to date
    • change log entries
  • Testing:
    • Unit tests
    • System (end to end) tests
    • Manual testing
  • API is compatible with existing add-ons.
  • Documentation:
    • User Documentation
    • Developer / Technical Documentation
    • Context sensitive help for GUI changes
  • UX of all users considered:
    • Speech
    • Braille
    • Low Vision
    • Different web browsers
    • Localization in other languages / culture than English

@AppVeyorBot
Copy link

  • Build execution time has reached the maximum allowed time for your plan (60 minutes).

See test results for failed build of commit 61e47fbed5

@michaelDCurran

This comment has been minimized.

Co-authored-by: Łukasz Golonka <lukasz.golonka@mailbox.org>
@AppVeyorBot

This comment has been minimized.

@AppVeyorBot

This comment has been minimized.

@AppVeyorBot

This comment has been minimized.

…to its own Python function to allow unit tests to continue to pass. Unit tests do not require this library.
Copy link
Contributor

@feerrenrut feerrenrut left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are a few build warnings introduced, these may be outside your control, but it would be good if they were silenced:

"C:\work\repo\nvda\PR\include\microsoft-ui-uiautomation\src\uiAutomation\microsoft.ui.uiautomation\Microsoft.UI.UIAutomation.vcxproj" (Build target) (1) ->
(PrepareForBuild target) ->
  C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\MSBuild\Microsoft\VC\v160\Microsoft.CppBuild.targets(499,5): warning MSB8004: Output Directory does not end with a trailing slash.  This build instance will add the slash as it is required to allow proper evaluation of the Output Directory. [C:\work\repo\nvda\PR\include\microsoft-ui-uiautomation\src\uiAutomation\microsoft.ui.uiautomation\Microsoft.UI.UIAutomation.vcxproj]

nvdaHelper/UIARemote/sconscript Show resolved Hide resolved
nvdaHelper/UIARemote/remoteLog.h Outdated Show resolved Hide resolved
nvdaHelper/UIARemote/UIARemote.cpp Outdated Show resolved Hide resolved
nvdaHelper/UIARemote/UIARemote.cpp Outdated Show resolved Hide resolved
nvdaHelper/UIARemote/UIARemote.cpp Outdated Show resolved Hide resolved
nvdaHelper/UIARemote/UIARemote.cpp Outdated Show resolved Hide resolved
nvdaHelper/UIARemote/UIARemote.cpp Outdated Show resolved Hide resolved
…VT_HEADER_DEPRECATION_WARNING` by converting the const char* from exception::what to wchar_t using multibyteToWideChar as recommended by the compiler warning, rather than using wstring_convert deprecated in c++17.
…g UiaArray to a local vector, and also ensure it is not copied.
…{outDir} passed to msbuild to suppress warnings amout the missing slash.
@michaelDCurran
Copy link
Member Author

@feerrenrut I believe I have addressed all your points.

nvdaHelper/UIARemote/UIARemote.cpp Outdated Show resolved Hide resolved
nvdaHelper/UIARemote/UIARemote.cpp Show resolved Hide resolved
nvdaHelper/UIARemote/UIARemote.cpp Outdated Show resolved Hide resolved
nvdaHelper/UIARemote/remoteLog.h Outdated Show resolved Hide resolved
nvdaHelper/readme.md Outdated Show resolved Hide resolved
nvdaHelper/readme.md Outdated Show resolved Hide resolved
nvdaHelper/readme.md Outdated Show resolved Hide resolved
nvdaHelper/readme.md Outdated Show resolved Hide resolved
michaelDCurran and others added 6 commits February 16, 2022 02:13
…to its own utility function.

also ensure that the buffer that receives the wide chars is initialized to 0s and is large enough to allow for a null-terminator.
Fix caps and typos.

Co-authored-by: Reef Turner <feerrenrut@users.noreply.github.com>
Copy link
Contributor

@feerrenrut feerrenrut left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll leave it up to you about this, I think the C++ standard library approach is less error prone and sets a good example.

nvdaHelper/UIARemote/remoteLog.h Outdated Show resolved Hide resolved
@AppVeyorBot
Copy link

See test results for failed build of commit bf6ff0d7ba

@feerrenrut
Copy link
Contributor

The test failure looks unrelated. I have kicked off the build again.
Old build link: https://ci.appveyor.com/project/NVAccess/nvda/builds/42592615/tests

@michaelDCurran michaelDCurran merged commit a3ed8b8 into master Feb 16, 2022
@michaelDCurran michaelDCurran deleted the msWordCustomAttributeValues branch February 16, 2022 10:31
@nvaccessAuto nvaccessAuto added this to the 2022.1 milestone Feb 16, 2022
michaelDCurran added a commit that referenced this pull request Feb 16, 2022
…r, layout column number and section number (#13283)"

This reverts commit a3ed8b8.
michaelDCurran added a commit that referenced this pull request Feb 16, 2022
…r, layout column number and section number (#13283)" (#13350)

This reverts commit a3ed8b8.
Reverted due to #13346 and microsoft/microsoft-ui-uiautomation/#89
michaelDCurran added a commit that referenced this pull request Feb 23, 2022
…t column number and section number (#13283)

Supersedes pr #13149

Summary of the issue:
The following error is in the log when viewing annotations in the NVDA elements list:
```
ERROR - queueHandler.flushQueue (19:47:20.811) - MainThread (9476):
Error in func ElementsListDialog.initElementType
Traceback (most recent call last):
  File "queueHandler.pyc", line 55, in flushQueue
  File "browseMode.pyc", line 1054, in initElementType
  File "browseMode.pyc", line 1074, in filter
  File "NVDAObjects\UIA\wordDocument.pyc", line 139, in label
  File "NVDAObjects\UIA\wordDocument.pyc", line 95, in getCommentInfoFromPosition
AttributeError: 'WordBrowseModeDocument' object has no attribute '_UIACustomAnnotationTypes'
```
This is because getCommentInfoFromPosition tries to fetch custom annotation types off its obj property, which it incorrectly expects to be a UIA NVDAObject. But in this case it is a TreeInterceptor.
PR #13149 was opened to address this by specifically creating a UIA NVDAObject rather than getting it from the position TextInfo. However, Further investigation has found that it is actually not necessary to even support custom annotation types (draft comment, resolved comment) in getCommentInfoFromPosition as:
1. Quicknav / Elements List iteration was never updated to specifically jump to / list those custom comment types. See CommentUIATextInfoQuickNavItem.wantedAttribValues
2. Although the IUIAutomationTextRange implementation in MS Word does return these custom comment types in GetAttributeValue when given UIA_AnnotationTypesAttributeId, the elements returned with UIA_AnnotationObjectsAttributeId only ever return the standard Comment annotation type via their UIA_AnnotationAnnotationTypeId property. In fact for draft comments, no annotation objects are returned at all.
In short, supporting custom comment annotation types in getCommentInfoFromPosition was not necessary in the first place.

Description of how this pull request fixes the issue:
Remove support for custom comment annotation types from getCommentInfoFromPosition. It again only supports the standard comment annotation type.
michaelDCurran added a commit that referenced this pull request Feb 24, 2022
…Word via UIA, using UIA remote ops (#13387)

This reintroduces the commits in pr #13283 which had been reverted in pr #13350. However it also addresses build issues.

Summary of the issue:
PR #13283 introduced support for reporting line/column/section numbers in Microsoft word via UI Automation, by using the UI Automation Remote Operations Library.
However, this PR had to be reverted in #13350 as
1. UIA would not initialize properly in binary builds due to a missing manifest file. (Improper initialization of UIA during NVDA start #13347)
2. NVDA failed to build on Visual Studio 2022. Specifically when building the Remote Ops library with msbuild, midl would fail with an error about a system environment variable being missing.
3. NVDA could no longer be built on Windows 7. (Impossible to build NVDA after introduction of UIA remote library #13346)

Description of how this pull request fixes the issue:
This PR reintroduces all of the original changes, but:
• setup.py now includes *.manifest files in the lib directory.
• NVDA again now requires Visual Studio 2019. However, builds will not fail if a newer version of Visual Studio (E.g. 2022) is installed along side 2019. this is managed by setting MSVC_VERSION in scons before it looks for Visual Studio, so that it specifically selects VS 2019 (VC 14.2).
Although building on Windows 7 could not be fixed, the readme now notes that only building on Windows 10 and higher is supported.
@LeonarddeR LeonarddeR mentioned this pull request Mar 22, 2022
5 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants