Support math content in MS Word build 14326+ via UIA by michaelDCurran · Pull Request #12977 · nvaccess/nvda

michaelDCurran · 2021-10-22T03:50:17Z

Link to issue number:

Summary of the issue:

NVDA was able to provide rich support of mathematical content in Microsoft Word by making use of MathPlayer, MathType and the MS Word object model.
However, when NVDA only relies on UI Automation to access MS Word docuents, it is no longer possible to fetch mathMl from MathType equations via the MS Word object model.
However, Microsoft very recently added a new UI Automation custom property for fetching raw mathMl from an Office Equation node, thus if NVDa were to fetch this mathMl, then it could pass it along to mathPlayer as usual, then providing rich reading / navigation of math content in MS word again.

Description of how this pull request fixes the issue:

Register the mathMl UIA custom property.
Fetch the raw mathMl from an MS Word Office Equation node.
Expose it for the rest of NVDA to find and use with mathPlayer.

Note that MS Word also provides a basic form of linea symbol navigation in equations, so we make sure that the mathMl is exposed to mathPlayer when moving by line, replacing MS Word's own inner math content. But if moving by word/character, then MS Word's own inner math content is left in tact.

Testing strategy:

Opened an MS Word document
Started typing a math equation by pressing alt+=
Typed an equation like x^2+x+1 and pressed enter.
Pressed up arrow to move back up to the equation, confirming that NVDA read out the equation with mathPlayer, and also brailled the equation in Nemeth code.
Used the left and right arrow keys to move through the equation by symbol. Confirming that Microsoft Word's own navigation was used, and each symbol was reported.
Pressed NvDA+alt+m to interact with the Math using MathPlayer and moved aorund the math tree with the arrows.
Pressed escape to exit Math interaction mode.
Switched to Browse mode with NVDA+space.
Pressed enter to activate the current item in browse mode, ensuring that mat interaction mode was entered.
Pressed ecape to exit math interaction mode.
Still in browse mode, pressed NvDA+alt+m to enter math interaction mode.
Pressed escape to exit math interaction mode.

Known issues with pull request:

Users / authors will have to convert any old MathType equations to Office Equations by selecting each equation, and then choosing Equation Options -> Convert to Office math in the context menu.
The experience for typing math equations is now slightly different in that the user needs to now use Microsoft Word's own math input feature. I.e. pressing alt+= and typing the equation. However, MS Word does now support several input syntaxes, including LateX similar to MathType.

Change log entries:

New features
In Microsoft Word accessed via UI Automation, NVDA will now make use of mathPlayer to read and navigate Office math equations. (#12946)
- For this to work, you must be running Microsoft Word 365/2016 build 14326 or later.
- MathType equations must also be manually converted to Office Math by selecting each and choosing Equation options -> Convert to Office Math in the context menu.
Changes
Bug fixes
For Developers

Code Review Checklist:

Pull Request description:
- description is up to date
- change log entries
Testing:
- Unit tests
- System (end to end) tests
- Manual testing
API is compatible with existing add-ons.
Documentation:
- User Documentation
- Developer / Technical Documentation
- Context sensitive help for GUI changes
UX of all users considered:
- Speech
- Braille
- Low Vision
- Different web browsers
- Localization in other languages / culture than English

michaelDCurran · 2021-10-22T03:52:06Z

Assuming that we keep accessing UIA by default in recent builds of MS Word for NvDA 2021.4, it is necessary for this pr to be also included, otherwise users will loose access to mathematical content.

AppVeyorBot · 2021-10-22T04:09:58Z

PASS: Translation comments check.
PASS: Unit tests.
FAIL: Lint check. See test results for more information.
PASS: System tests.
Build (for testing PR): https://ci.appveyor.com/api/buildjobs/q03ukn2o5d3o24h4/artifacts/output/nvda_snapshot_pr12977-24074,44ef7b49.exe

See test results for failed build of commit 44ef7b49e4

source/NVDAObjects/UIA/wordDocument.py

seanbudd · 2021-10-22T04:46:25Z

source/NVDAObjects/UIA/wordDocument.py

 class WordBrowseModeDocument(UIABrowseModeDocument):

-	def shouldSetFocusToObj(self,obj):
+	def _shouldSetFocusToObj(self, obj: NVDAObject) -> bool:


Is changing the name of this function an API breaking change?

No, it was actually incorrectly named in the first place. It should be an override for BrowseModeDocumentTreeInterceptor._shouldSetfocusToObj. In other words, that code never actually ran. It correctly does now.

However, there may be add-ons that are calling shouldSetFocusToObj directly. If this is going into 2021.3 then I think there should be an alias (then removed in 2022.1)

def _shouldSetFocusToObj(self, obj: NVDAObject) -> bool: ... if nvdaversion.year < 2022: shouldSetFocusToObj = _shouldSetFocusToObj # deprecated to be removed in 2022.1

Qchristensen

This is good and will be appreciated by users, great work :)

cary-rowen · 2021-10-22T07:46:59Z

Oh, cool!

LeonarddeR · 2021-10-22T08:55:54Z

source/NVDAObjects/UIA/wordDocument.py

+		elif obj.role == controlTypes.Role.MATH:
+			# Don't set focus to math equations otherwise they cannot be interacted with mathPlayer.
+			return False
 		return super(WordBrowseModeDocument,self).shouldSetFocusToObj(obj)


Don't forget this one

Suggested change

return super(WordBrowseModeDocument,self).shouldSetFocusToObj(obj)

return super()._shouldSetFocusToObj(obj)

LeonarddeR · 2021-10-22T08:58:38Z

source/NVDAObjects/UIA/wordDocument.py


-	def shouldSetFocusToObj(self,obj):
+	def _shouldSetFocusToObj(self, obj: NVDAObject) -> bool:
 		# Ignore strange editable text fields surrounding most inner fields (links, table cells etc) 


Could you move the below lines into a helper function and call that in both _shouldSetFocusToObj and shouldPassThrough? That way it's easier to see equal logic is used.

I don't think this is worth doing as it is really just two lines in each. An elif and a return False. Also, just because the lines do seem to look the same, they are of course representing slightly different questions.

LeonarddeR · 2021-10-22T08:59:55Z

source/NVDAObjects/UIA/wordDocument.py

+		try:
+			return self._getUIACacheablePropertyValue(self._UIACustomProps.word_mathml.id)
+		except COMError:
+			pass


Rather than passing (i.e. returning None implicitly), I think it's nicer to return None explicitly here.

LeonarddeR · 2021-10-22T09:01:43Z

user_docs/en/userGuide.t2t

+Note however that any previously created MathType equations must be  first converted to Office Math by selecting each and choosing Equation Options -> Convert to Office Math in the context menu. Ensure your version of MathType is the latest version before doing this.
+Microsoft Word also now provides linea symbol-based navigation through the equations itself, and supports inputting math using several syntaxes, including LateX. For further details, please see [Linear format equations using UnicodeMath and LaTeX in Word https://support.microsoft.com/en-us/office/linear-format-equations-using-unicodemath-and-latex-in-word-2e00618d-b1fd-49d8-8cb4-8d17f25754f8]
+- Microsoft Powerpoint, and older versions of Microsoft Word: 
+NVDA can read and navigate MathType equations in both Microsoft Powerpoint and Microsoft word.


Just to make sure, have you verified that this is formatted nicely?

user_docs/en/userGuide.t2t

… should have always been _shouldSetFocusToObj

Co-authored-by: Sean Budd <sean@nvaccess.org>

michaelDCurran · 2021-10-26T02:35:54Z

This has been rebased to master and will be not in NVDA 2021.3.

cary-rowen · 2021-10-26T02:40:25Z

It's a shame not to include this pr in 2021.3.

CyrilleB79 · 2021-10-26T07:06:49Z

@cary-rowen please note also that UIA will not be forced by default in 2021.3 according to #12989. So this shouldn't be a problem not to have this PR included.

codeofdusk · 2021-11-25T04:19:05Z

This PR is still listed under 2021.3 changes. Is this a mistake?

…21.3 to 2022.1; the PR (nvaccess#12977) has been retargetted from beta to master but the change log item has been forgotten.

CyrilleB79 · 2021-11-25T08:56:33Z

Yes it's a mistake due to retargetting from beta to master. Good catch!
I have submitted #13095 to fix this.

…2022.1; the PR (#12977) has been retargetted from beta to master but the change log item has been forgotten. (#13095)

michaelDCurran requested review from a team as code owners October 22, 2021 03:50

michaelDCurran requested review from Qchristensen and seanbudd and removed request for a team October 22, 2021 03:50

michaelDCurran added this to the 2021.3 milestone Oct 22, 2021

seanbudd reviewed Oct 22, 2021

View reviewed changes

seanbudd previously approved these changes Oct 22, 2021

View reviewed changes

Qchristensen previously approved these changes Oct 22, 2021

View reviewed changes

LeonarddeR suggested changes Oct 22, 2021

View reviewed changes

michaelDCurran dismissed stale reviews from Qchristensen and seanbudd via bda90f2 October 23, 2021 08:58

michaelDCurran force-pushed the i12946 branch from 6fc89e3 to bda90f2 Compare October 23, 2021 08:58

seanbudd reviewed Oct 25, 2021

View reviewed changes

user_docs/en/userGuide.t2t Outdated Show resolved Hide resolved

user_docs/en/userGuide.t2t Show resolved Hide resolved

user_docs/en/userGuide.t2t Outdated Show resolved Hide resolved

seanbudd approved these changes Oct 26, 2021

View reviewed changes

michaelDCurran and others added 7 commits October 26, 2021 12:32

Support mathML in MS Word via UIA.

08faeb0

Update User guide and changes.

824344d

Linting

ede3bc3

WordBrowseModeDocument: deprecate shouldSetFocusToObj in 2022.1 -- it…

c696d6e

… should have always been _shouldSetFocusToObj

Review actions.

1630b6b

Fix formatting of userGuide

8c7945c

Co-authored-by: Sean Budd <sean@nvaccess.org>

Word with UIA: remove deprecation for 2022.1.

80606f4

michaelDCurran changed the base branch from beta to master October 26, 2021 02:34

michaelDCurran force-pushed the i12946 branch from 4ae3b10 to 80606f4 Compare October 26, 2021 02:35

michaelDCurran removed this from the 2021.3 milestone Oct 26, 2021

michaelDCurran added this to the 2022.1 milestone Oct 26, 2021

michaelDCurran merged commit 9f8fff8 into master Oct 27, 2021

michaelDCurran deleted the i12946 branch October 27, 2021 03:01

CyrilleB79 mentioned this pull request Nov 25, 2021

Change log: move the change log item for issue #12946 from 2021.3 to 2022.1 section #13095

Merged

5 tasks

michaelDCurran pushed a commit that referenced this pull request Nov 25, 2021

Change log: move the change log item for issue #12946 from 2021.3 to …

1cb7e9e

…2022.1; the PR (#12977) has been retargetted from beta to master but the change log item has been forgotten. (#13095)

	return super(WordBrowseModeDocument,self).shouldSetFocusToObj(obj)
	return super()._shouldSetFocusToObj(obj)

Uh oh!

Conversation

michaelDCurran commented Oct 22, 2021

Link to issue number:

Summary of the issue:

Description of how this pull request fixes the issue:

Testing strategy:

Known issues with pull request:

Change log entries:

Code Review Checklist:

Uh oh!

michaelDCurran commented Oct 22, 2021

Uh oh!

AppVeyorBot commented Oct 22, 2021

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Qchristensen left a comment

Choose a reason for hiding this comment

Uh oh!

cary-rowen commented Oct 22, 2021

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

michaelDCurran commented Oct 26, 2021

Uh oh!

cary-rowen commented Oct 26, 2021

Uh oh!

CyrilleB79 commented Oct 26, 2021

Uh oh!

codeofdusk commented Nov 25, 2021

Uh oh!

CyrilleB79 commented Nov 25, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants