Integrate support for MathPlayer

[nvaccessAuto]

Reported by jteh on 2014-12-08 04:02
The new MathPlayer library from Design Science for math accessibility will be released as a free download soon. Since it will be available to everyone and it has advantages over pyMathSpeak, we can use this as our official math implementation, at least for now. It will still be possible to replace this with add-ons if desired.

Practically speaking, this means adding support to the core. MathPlayer still has to be installed separately, but it will work with NVDA once installed.

Most of this work has already been done, but it needs to be ported to the core math framework ( #4509).
Blocked by #4509, #4674

[nvaccessAuto]

Comment 7 by camlorn on 2015-03-05 19:45
I'm playing with the beta:
The fact that a settings panel is available is not very obvious. If Mathplayer is installed, perhaps consider adding a preferences menu item that launches it from NVDA. At the moment, this involves going into control panel and knowing that it's likely to exist. I cannot seem to trigger the Mathplayer context menu in any way.
There seem to be no pauses at all in anything anywhere. I'm not sure if this is a bug or not or if there's a setting or what, but it all just blurs together.
And finally, in Firefox and with pages that use MathJax, it appears to break interestingly: if mathjax isn't set to render mathml, you get "mathml error" until you fix it.
If there is a more appropriate place to send feedback on broken MathML constructs, directing users from this ticket and the official blog post to such would perhaps be useful.

[nvaccessAuto]

Comment 8 by derrickf on 2015-03-11 22:54
I've been using the NVDA reader (for math) + MathPlayer 4 beta, and Firefox v36.0.1. It reads the mathml equations on this page http://www.w3.org/Math/XSL/csmall2.xml in my current working environment (Win 7 32-bit). Another user is attempting to read the same page using the NVDA reader portable copy (for math), but is getting an error when trying to read the page. This is the error he is seeing:

Input: kb(laptop):downArrow
ERROR - scriptHandler.executeScript (13:16:05):
error executing script: <bound method Gecko_ia2.script_moveByLine_forward of <virtualBuffers.gecko_ia2.Gecko_ia2 object at 0x05AB3AF0>> with gesture u'down arrow'
Traceback (most recent cal last):
File "scriptHandler.pyc", line 176, in executeScript
File "cursorManager.pyc", line 191, in script_moveByLine_forward
File "cursorManager.pyc", line 124, in caretMovementScriptHelper
File "speech.pyc", line 728, in speakTextInfo
File "speech.pyc", line 613, in speakTextInfo_addMath
File "virtualBuffers__init.pyc", line 381, in getMathMl
File "baseObject.pyc", line 34, in get
File "baseObject.pyc", line 110, in getPropertyViaCache
File "NVDAObjects\IAccessible\mozilla.pyc", line 179, in get_mathMl
File "comtypes__init.pyc", line 1078, in QueryInterface
COMError: (-2147467262, 'No such interface supported', (None, None, None, 0, None))

Could this error possibly be occurring because he's using a portable copy, or possibly that he is not using the NVDA reader portable copy (for math), but using a different NVDA build? I will try the NVDA portable copy (for math) to verify it works with the test page.

update I installed the NVDA portable copy (for math), and it reads the mathml equations

[nvaccessAuto]

Comment 9 by jteh (in reply to comment 7) on 2015-03-12 04:31
Replying to camlorn:

The fact that a settings panel is available is not very obvious.

IMO, this should be covered in MathPlayer documentation. I guess I'm not totally against opening it from NVDA's GUI. However, most users will have to read the MathPlayer documentation anyway to work out how navigation works, etc., and NVDA's User Guide specifically notes that a user should consult MathPlayer's documentation. Of course, right now, there doesn't seem to be well structured MathPlayer documentation, but I'd hope that gets resolved before the final release.

There seem to be no pauses at all in anything anywhere. I'm not sure if this is a bug or not or if there's a setting or what, but it all just blurs together.

We discussed this elsewhere, but for the benefit of others, this won't yet work with speechPlayerInEspeak (Edward).

And finally, in Firefox and with pages that use MathJax, it appears to break interestingly: if mathjax isn't set to render mathml, you get "mathml error" until you fix it.

Ug. It looks like MathJax is setting role="math" on its root nodes, even though you can't access any useful math from there. I'll try to tweak things so we at least don't throw errors in that case, but until MathJax implement a way for ATs to get at MathML, you won't get any useful math.

[nvaccessAuto]

Comment 10 by jteh (in reply to comment 8) on 2015-03-12 04:34
Replying to derrickf:

File "NVDAObjects\IAccessible\mozilla.pyc", line 179, in _get_mathMl

File "comtypes__init__.pyc", line 1078, in QueryInterface

COMError: (-2147467262, 'No such interface supported', (None, None, None, 0, None))

It looks like the Firefox DOM APIs aren't registered. Is he using portable Firefox? That won't work. Note that we only use these APIs for math, so everything else will work as expected.

[nvaccessAuto]

Comment 11 by jteh (in reply to comment 10) on 2015-03-12 04:36
Replying to jteh:

It looks like the Firefox DOM APIs aren't registered.

Actually, that might not be correct; that'd probably be a different error. Hmm. In that case, I'm not sure what's going on here.

[nvaccessAuto]

Comment 12 by driemer.riemer@... on 2015-03-12 18:07
I get the same error he is getting both in firefox and ie. I upgraded from the previous version of mathplayer, and have mathtype installed. I don't know what needs done to boot mathplayer anywho. I assumed it would just work when I come acrossed math.

[nvaccessAuto]

Comment 13 by driemer.riemer@... on 2015-03-15 22:31
I have two more errors, these while navigating in word.
Something seems broken with mathplayer, but the desci support couldn't tell me what to do. Mathml pages in firefox and ie seem to trigger a com error as well.
Braille regions text: linearly dependent if and only if one of the vectors is a multiple of the other. And any set containing the zero vector is '
IO - braille.BrailleHandler.update (16:25:37):
Braille window dots: 24 234 - 123 24 1345 15 1 1235 123 13456 - 145 15 1234 15 1345 145 15 1345 2345 - 24 124 - 1 1345 145 - 135 1345 123 13456 - 24 124 -
IO - braille.BrailleHandler.update (16:25:37):
Braille window dots: 24 234 - 123 24 1345 15 1 1235 123 13456 - 145 15 1234 15 1345 145 15 1345 2345 - 24 124 - 1 1345 145 - 135 1345 123 13456 - 24 124 -
ERROR - scriptHandler.executeScript (16:25:37):
error executing script: <bound method Dynamic_IAccessibleWordDocumentWindowNVDAObject.script_caret_moveByLine of <NVDAObjects.Dynamic_IAccessibleWordDocumentWindowNVDAObject object at 0x05B55A50>> with gesture u'up arrow'
Traceback (most recent call last):
File "scriptHandler.pyc", line 176, in executeScript
File "editableText.pyc", line 131, in script_caret_moveByLine
File "editableText.pyc", line 103, in _caretMovementScriptHelper
File "editableText.pyc", line 89, in _caretScriptPostMovedHelper
File "speech.pyc", line 806, in speakTextInfo
File "speech.pyc", line 613, in _speakTextInfo_addMath
File "mathPres\mathPlayer.pyc", line 137, in getSpeechForMathMl
File "mathPres\mathPlayer.pyc", line 51, in _processMpSpeech
TypeError: expected string or buffer
ERROR - core.Notify (16:25:37):
errors in this core pump cycle
Traceback (most recent call last):
File "core.pyc", line 346, in Notify
File "braille.pyc", line 1548, in pumpAll
File "braille.pyc", line 1476, in handlePendingCaretUpdate
File "braille.pyc", line 1482, in doCursorMove
File "braille.pyc", line 822, in update
File "braille.pyc", line 745, in addTextWithFields
File "textInfos__init.pyc", line 431, in getControlFieldBraille
File "braille.pyc", line 621, in getControlFieldBraille
File "mathPres\mathPlayer.pyc", line 142, in getBrailleForMathMl
COMError: (-2147467259, 'Unspecified error', (None, None, None, 0, None))
IO - inputCore.InputManager.executeGesture (16:25:38):
Input: kb(laptop):NVDA+f1

[nvaccessAuto]

Comment 14 by jteh (in reply to comment 12) on 2015-03-15 23:06
Replying to driemer.riemer@…:

I get the same error he is getting both in firefox and ie.

It's actually impossible to get the traceback from comment:8 in IE. If this is a different traceback to the one you posted in comment:13, please check this again.

I don't know what needs done to boot mathplayer anywho. I assumed it would just work when I come acrossed math.

It should.

[nvaccessAuto]

Comment 15 by jteh (in reply to comment 13) on 2015-03-15 23:12
Replying to driemer.riemer@…:

I have two more errors, these while navigating in word.

Have you been able to successfully access any math at all? Are you able to provide the Word document you're working with?

File "mathPres\mathPlayer.pyc", line 137, in getSpeechForMathMl

File "mathPres\mathPlayer.pyc", line 51, in _processMpSpeech

TypeError: expected string or buffer

Looks like MathPlayer is returning no spoken text for some reason. Ug.

Can you please try the following line in the Python console:

import mathPres; mathPres.ensureInit(); mathPres.speechProvider.getSpeechForMathMl("<math><mi>x</mi></math>")

Do you get anything?

[nvaccessAuto]

Comment 16 by driemer.riemer@... (in reply to comment 15) on 2015-03-16 01:18
Replying to jteh:

Replying to driemer.riemer@…:

I have two more errors, these while navigating in word.

Have you been able to successfully access any math at all? Are you able to provide the Word document you're working with?

I can maybe privatly provide a snippett of it, it is a math book for linear algebra, I would need to email it to either you or mik. I get no math output absolutely anywhere with mathplayer.

File "mathPres\mathPlayer.pyc", line 137, in getSpeechForMathMl

File "mathPres\mathPlayer.pyc", line 51, in _processMpSpeech

TypeError: expected string or buffer

Looks like MathPlayer is returning no spoken text for some reason. Ug.

Can you please try the following line in the Python console:

import mathPres; mathPres.ensureInit(); mathPres.speechProvider.getSpeechForMathMl("<math><mi>x</mi></math>")

Do you get anything?

Oddly I get this traceback.
Traceback (most recent call last):
File "", line 1, in
File "mathPres\mathPlayer.pyc", line 137, in getSpeechForMathMl
File "mathPres\mathPlayer.pyc", line 51, in _processMpSpeech
TypeError: expected string or buffer

[nvaccessAuto]

Comment 17 by jteh on 2015-03-16 03:56
Oh dear. That's definitely broken.

What about this?

import mathPres; mathPres.ensureInit(); mathPres.speechProvider._mpSpeech.SetMathML("<math><mi>x</mi></math>"); mathPres.speechProvider._mpSpeech.GetSpokenText()

Also, please check the following:

1. Open Control Panel.
2. Open the Control Panel applet for MathPlayer. (Tip: Type math into the search box.)
3. What is the language set to? Try playing with that.

Finally, try reinstalling MathPlayer if you haven't already.

[nvaccessAuto]

Comment 18 by driemer.riemer@... on 2015-03-16 04:15
That code seems to have returned None
Also, Do you know has anyone tested this on win 7 with both mathtype (whatever the version that is current is) and mathplayer? I might be able to test it on my 8.1 vm for compiling nvda with, since I kind of am in bad need of a reformat on this machine (I have not reformatted in 2 years).

[nvaccessAuto]

Comment 19 by jteh on 2015-03-16 04:57
I didn't test with MathType in Windows 7, but I did test MathPlayer with browsers.

[nvaccessAuto]

Comment 20 by Palacee_hun on 2015-03-17 20:53
I would have wanted to beta-test math support on my Windows XP SP3 32-bit box, so I went ahead with installing MathPlayer 4 public beta. After the installer had extracted files, it popped up an error dialog with a criptic message. It complained that it could not run a file called setup.exe from the temp folder.
[contacted the tech support at Designscience, but haven't got a reply yet. Meanwhile I looked into this myself, and found out that the PE header of this problematic setup.exe says "OS version = 6.0", that is above WinXP.
[[BR]([BR]]
I)]
Is it true that MathPlayer 4 doesn't run on WinXP? My findings tell me so, however
I've found no place where this is stated.
[[BR]]
Does anybody know anything definite about this? If this is so, then this should at least be documented and an appropriate error message displayed, or better still mathPlayer developers convinced that WinXP should be supported until NVDA supports it. As far as I know a lot of vi folks still use WinXP in some countries where NVDA is popular as a free screen reader. And a Windows upgrade is not a walk in the park to say the least.

[nvaccessAuto]

Comment 21 by jteh on 2015-03-18 04:18
I've just asked my contact at Design Science. He confirmed that MathPlayer 4 does not support Windows XP and they have no plans to do so. They had problems on XP, and given that XP is now end-of-life, they aren't willing to invest resources into resolving these. That said, you're correct that this is not documented and they definitely need to address this.

[nvaccessAuto]

Comment 22 by Palacee_hun on 2015-03-18 11:48
I also got confirmation from DesignScience tech support. I understand, but of course I am not very happy.

[nvaccessAuto]

Comment 23 by jteh (in reply to comment 18) on 2015-04-13 02:29
Replying to driemer.riemer@…:

That code seems to have returned None

Any luck with this? Did you try my other suggestions from comment:17? Otherwise, I suggest you contact Design Science. As far as I can tell, there is something wrong with MathPlayer on your system.

[nvaccessAuto]

Comment 24 by driemer.riemer@... on 2015-04-13 14:33
For whatever reason none of 100 uninstalls worked, but then I wiped nvda, wiped firefox, wiped mathplayer with revo uninstall to get it totally out of my registry, and then reinstalled and it worked fine. I did manage to get it to refuse to show braille on some ridiculous looking linear algebra, which I could probably share with you or mik privately so you could see what the error is. It looked like something with an IndexError.

[nvaccessAuto]

Comment 25 by jteh on 2015-04-15 06:33
Did you get spoken math even though braille failed?

If you can provide the exception, that might be enough. Otherwise, feel free to email me the test case privately. Thanks.

[nvaccessAuto]

Comment 26 by jteh (in reply to comment 9) on 2015-04-15 06:42
Replying to jteh:

And finally, in Firefox and with pages that use MathJax, it appears to break interestingly: if mathjax isn't set to render mathml, you get "mathml error" until you fix it.

... I'll try to tweak things so we at least don't throw errors in that case

Done in aaf3e71.

but until MathJax implement a way for ATs to get at MathML, you won't get any useful math.

This is being discussed in the MathJax project at the moment. See GitHubIssue:mathjax:MathJax:938.

[nvaccessAuto]

Comment 27 by James Teh <jamie@... on 2015-04-17 04:07
In [e4c7b9a]:

Merge branch 't4673' into next

Incubates #4673.

Changes:
Added labels: incubating

[nvaccessAuto]

Comment 28 by bdorer on 2015-04-26 19:09
Hi, on Wikipedia I found the following line:
\mathrm{^\circ C}
is this Math? NVDA reported this isn't. I used Firefox 39 with NVDA next.

[nvaccessAuto]

Comment 29 by jteh on 2015-04-26 23:27
It is math, but it's a LaTeX alt tag on a graphic. As noted in the NVDA User Guide, only proper MathML is supported. However, if you have/create a Wikipedia account, you can set Wikipedia to use MathML in the math preferences for your account.

[nvaccessAuto]

Comment 30 by Q on 2015-04-27 02:19
If we wanted to actually parse this and make it available to MathPlayer it looks like we could use something like https://pypi.python.org/pypi/mathdom/

from mathml.lmathdom import MathDOM # use lxml implementation
doc = MathDOM.fromString("+2^x+4*-5i/6","infix_term")   # parse infix term

[nvaccessAuto]

Comment 31 by jteh on 2015-04-27 02:29
That looks like a very useful module; thanks for the link. I had looked at other ways of handling conversions in the past, but didn't come across this one.

The problem is that there's no standard/consistent way to identify that something is a LaTeX alt tag. We don't even know if it's math. There are heuristics we could use to guess this some of the time (e.g. looking for common LaTeX commands), but it's not going to be reliable and it could also potentially be very slow. It's also difficult to know where this "guessing" code would even fit architecturally speaking.

Note that much of the need for this should go away once MathJax implements accessibility support; see comment:26. Also, I think the hope is that this MathML mode would be enabled by default in Wikipedia eventually.

Even if we do end up doing something like this, it won't be part of the initial implementation and should thus be handled in a separate ticket.

[nvaccessAuto]

Comment 32 by James Teh <jamie@... on 2015-05-01 05:03
In [7bcc9e6]:

Using MathPlayer 4 from Design Science, NVDA can now read and interactively navigate mathematical content in web browsers and in Microsoft Word and PowerPoint.

Fixes #4509, #4673, #4674.

Changes:
Removed labels: incubating
State: closed

[nvaccessAuto]

Comment 33 by jteh on 2015-05-01 05:18
Changes:
Milestone changed from None to 2015.2

[nvaccessAuto]

Comment 34 by driemer.riemer@... on 2015-05-27 07:01
Hey Jamie,
Did you git the email with a few fragments of math from a textbook that seem to break nvda's math support in odd ways? If so, will this ticket need reopened?

[nvaccessAuto]

Comment 35 by jteh (in reply to comment 34) on 2015-05-27 10:57
Replying to driemer.riemer@…:

Did you git the email with a few fragments of math from a textbook that seem to break nvda's math support in odd ways?

I did get the email; thanks. I haven't had a chance to look at the file yet, though.

If so, will this ticket need reopened?

No; we'll handle this separately. My hunch is that it's probably a MathPlayer bug (since other math does work). Given that this seems to be the exception rather than the norm, I don't want to delay the 2015.2 release for this, especially because it is already significantly overdue. That's not to say we won't look into this; it just means it won't get fixed for 2015.2.

[nvaccessAuto]

Comment 36 by driemer.riemer@... on 2015-05-27 16:37
Okay. I agree with your way of doing this, because most people aren't going to be doing linear algebra and math at that high of a level. This shouldn't hold off nvda's release since it may only affect a few users.