Optional experimental support for Microsoft Excel via UI Automation #12210
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Allow Excel to be used via UIA if NVDA cannot inject in-process or an option is set in NVDA's advanced settings.
Includes:
- Support for selections:
* Support for selectionContainer / getSelectedItemsCount
* Only supporting UIA's selectionPattern2 currently.
* Allow duplicate focus UIA events as selecting cells causes duplicate focus events.
- Support overflowing / cropped states on cells.
- Adds cell presentation command
* Triggered with NVDA+o.
- Cell information is presented to the user via a browseable message.
* Script to list call appearance information:
- Cell size
- Rotation
- Outline color and thickness
- Fill color and type
- The number format
- Whether data validation is present
- The data validation prompt
- Whether there is any conditional formatting
- If gridlines are visible
- Export hasFormula and hasPopup (data validation dropdown) states on cells.
Remove unused imports, convert to explicit imports. There are still remaining issues caused by the locations of the file comInterfaces/* files not matching the namespace they are imported from EG: 'from comtypes.gen.UIAutomationClient'
… an annotation type of comment. Needed mostly to report existance of comments in Microsoft Excel.
…r and circular reference error messages in the cell's description. As part of this, a UIAAnnotationObjects property was added to the base UIA NVDAObject.
…er author is currently editing the cell.
|
@feerrenrut There are several more commits for you to review which were added after your last rebase of this branch. These comments add further functionality exposed by existing standard UI Automation annotations, and completes all possible support we can do with what Microsoft has currently exposed. |
This comment has been minimized.
This comment has been minimized.
|
The option to use UIA for Excel should be documented in the User Guide. Don't forget to add context help to it. |
|
Nice to see that this is finally PR ready after nearly two years of work! |
feerrenrut
suggested changes
Mar 23, 2021
Co-authored-by: Reef Turner <feerrenrut@users.noreply.github.com>
LeonarddeR
suggested changes
Mar 23, 2021
Co-authored-by: Reef Turner <feerrenrut@users.noreply.github.com>
This comment has been minimized.
This comment has been minimized.
michaelDCurran
force-pushed
the
excelWithUIA-rebase
branch
from
9dbf0d0
to
3f1d86b
Compare
CyrilleB79
reviewed
Mar 24, 2021
Co-authored-by: Cyrille Bougot <cyrille.bougot2@laposte.net>
|
@feerrenrut I think you are still requesting changes on this pr... |
See test results for failed build of commit b2ac158014 |
feerrenrut
approved these changes
Mar 24, 2021
|
I've now marked as approved |
CyrilleB79
reviewed
Mar 24, 2021
Co-authored-by: Cyrille Bougot <cyrille.bougot2@laposte.net>
|
I will file a follow up pr for support of merged cells. |
Ugh, this looked easy since merged cells are reflected in their row and column span, but it looks like it's not possible to get address and value that easy. Even when you include a merged cell in a selection, an error is generated: File "NVDAObjects\UIA\excel.py", line 370, in _get_cellCoordsText
firstSelected = sc.GetCurrentPropertyValue(
AttributeError: 'NoneType' object has no attribute 'QueryInterface' |
|
There is a problem with the layout of advanced related options .@michaelDCurran |
|
Thanks. Fixed in #12223. |
michaelDCurran
pushed a commit
that referenced
this pull request
Mar 25, 2021
The new option introduced by #12210 was not located in the "Microsoft UI Automation" option group as it should. Probably due to missed changes during upmerge of master. It is now Put in this group as intended by using The static box of the sizer as parent for the checkbox rather than the whole panel.
8 tasks
michaelDCurran
added a commit
that referenced
this pull request
Mar 26, 2021
With the merging of pr #12210 it became impossible to interact with UI Automation controls on Windows 7. * Fetching the annotationTypes UIA property raises a COMError if not implemented by the control. On newer Operating Systems UIA core returns a default. * Fetching the SelectionPattern2 interface also causes a COMError on older Operating Systems if not implemented by the control. Therefore: * Catch COMError when fetching UIA annotationTypes. * Catch COMError when fetching the UIA SelectionPattern2 interface. Fixes #12227
7 tasks
michaelDCurran
added a commit
that referenced
this pull request
Apr 7, 2021
…selectionContainer on UIA's SelectionItemPattern, such as for Lutlook Attachment lists. (#12274) After merging of pr #12210 , The attachment list in Outlook no longer read with NVDA, and an error was produced in the log. It seems that Outlook's attachment list, gives back a NULL pointer for the SelectionContainer property on its UIA SelectionItemPattern. This commit Checks to see if CurrentSelectionContainer is NULL and returns None as there is nothing further we can do with that pattern. Fixes #12265
michaelDCurran
added a commit
that referenced
this pull request
Apr 22, 2021
…as test code left over from pr #12210.
Merged
7 tasks
michaelDCurran
added a commit
that referenced
this pull request
Apr 22, 2021
… editing in cells (#12321) Fixes #12303 After merging of pr #12210 editing cells in Excel without UIA enabled became im possible as NVDA did not report / track focus had ented the Cell Edit control. This was due to the EXCEL6 window accidentally being marked as having a good UIA implementation. This was testing code left over from the early implementation of #12210. Description of how this pull request fixes the issue: Remove EXCEL6 from the good UIA windows list. Also ensure that MSAA focus events on this window are ignored when using Excel with UIA enabled, as Excel will fire its own UIA focus event on an edit control within the active cell.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Link to issue number:
None.
Summary of the issue:
Since 2013, Microsoft has been slowly adding a UI Automation implementation to Microsoft Excel. For quite some time, this implementation was missing too much to be at all useful for NVDA. However, in the last year, Microsoft has increased its effort in this area, starting to fill many of the gaps in the implementation, providing more of a reason for NVDA to start consuming this experimentally, with the goal of one day switching to UI Automation in Excel by default.
Not only would this provide significant performance improvemts for NVDA with Excel, but it would also enable scenarios such as using Excel in Application Guard, or remotely via Windows Virtual Desktop, where access to the object model and in-process injection is no longer possible.
Description of how this pull request fixes the issue:
NVDA will fall back to relying on UI Automation In Microsoft Excel if available, if NVDA is not able to inject in-process.
A new option has been added to NVDA's Advanced settings, which when enabled, forces NVDA to use UI Automation in Microsoft Excel always when available.
Although so far navigating, editing and querying cells is all supported, the following features are not yet implemented:
As some Excel-specific information on cells does not quite fit with standard NVDA concepts. A new Cell Appearance script (NVDA+o) has been added, which presents a browse mode document, listing these specific bits of information. This script may be changed or removed in future as it is possible to find this information out by looking in the Cell formatting dialog etc anyway, but for now it is there to demonstrate NVDa's ability to fetch this kind of information from UI Automation.
Many of Excel's specific features have been exposed by Microsoft through UI automation via either annotations, or custom properties.
Standard UIA annotations that NvDA consumes for this support are:
Microsoft has made use of UI Automation's custom property feature in order to expose many more Excel-specific properties: https://docs.microsoft.com/en-us/office/uia/excel/excelcustomproperties
This pr adds an infrastructure in UIAHandler / UIA NVDAObjects which allows registering and accessing these custom property IDs in a standard way.
The properties that Microsoft exposes in Excel are:
The above annotations and properties have only been available in Microsoft Excel since build 16.0.13522.10000.
Thus turning on this option will only be useful for this build or higher.
Testing strategy:
Manual testing with this spreadsheet which contains examples of each of the properties and annotations.
Known issues with pull request:
Change log entry:
What's new:
Code Review Checklist:
This checklist is a reminder of things commonly forgotten in a new PR.
Authors, please do a self-review and confirm you have considered the following items.
Mark items you have considered by checking them.
You can do this when editing the Pull request description with an x:
[ ]becomes[x].You can also check the checkboxes after the PR is created.