Should we drop MathML rendering?

[kevinbarabash]

The original purpose of MathML was to support a11y by having a version of the rendering that a screen reader could read. MathML is poorly supported by browsers and for the browser/screen reader combinations that do support it, what is read out isn't actually very helpful.

The solution we use on khanacademy.org is to generate text based on the KaTeX parse tree and then include that text on the page for the screen reader to read.

I'd like to make the following suggestions:

1. deprecated MathML generation code in KaTeX
2. stabilize the parse tree format (it's feels quite irregular, it'd be nice to fix that before people start depending on it's format)
3. include the text generation code in the contrib folder
4. figure out a translation strategy?

I'd love to hear what other people think about this.

[ronkok]

Murray Sargent has written some blog posts on this topic in the last few months. You might want to look into Math Accessibility Trees, Speaking of Math…, Nemeth Braille, Nemeth and Unicode, and Speaking Subscripts, Superscripts, and Fractions.

He believes there is more than one good way to write such a tree and he lays out some options.

[ronkok]

It would be great if the text generator could eventually be included in auto-render.min.js.

[gagern]

Just some ideas. Reading parts of the posts Ron referenced I get the impression that screen readers and Braille displays might be best considered independently. For the latter, providing access to the TeX input might well be a benefit for some users. Sometimes the TeX input may contain more information than the parsed tree. For example a\divides b conveys a semantic meaning which a\mid b does not, so expanding the former to the latter as part of macro expansion or parsing may have detrimental effects on the accessibility of the produced text.

[kevinbarabash]

Maybe we could tag certain items as semantic and then return the tree without expanding semantic commands. Other non-semantic commands could be dropped like horizontal spacing. There are other commands like \frac, \cfrac, \dfrac which are semantically the same. The code that generates that natural language version of text could map all of these to fraction though.

[edemaine]

Personally I'd love for it to at least be possible to turn off the MathML generation via an option.

[kevinbarabash]

@edemaine that's an option too, but if nobody's using there's no reason to keep it because it adds costs to future features.

[edemaine]

@kevinbarabash Good point. Here is more discussion by one of the leading math editors on Wikipedia: MathML considered harmful. He agrees that LaTeX source doesn't provide enough context to really infer the semantics, and that it's kinda useless as a presentation medium.

[kevinbarabash]

Cool article. Let's deprecate MathML then. As for accessibility, I think in the short term we can include the text generation code I mentioned earlier and update if the parse tree changes. I would like to minimize the number of things relying on the parse tree until its stable, but since we're already using this code at KA we'd have to update it anyways if the tree structure changed.

[edemaine]

Regarding your "stabilize the parse tree format", maybe a good starting point would be to try to document the existing parse tree format. I've certainly found it a challenge to work out from the code that uses it, and perhaps writing it down would make it clear how to make it more regular. For example, it seems "op" is pretty overloaded, e.g. it's the only way (?) to return a symbol from a function (even if the symbol isn't an operator) -- though perhaps that's more of an issue of how functions work -- and it seems to put them in the wrong font (?); see #599 (comment)

[xymostech]

In the original issue about adding accessibility to KaTeX ( #38 ) there were some strong proponents for producing MathML so I'd like to at least get some feedback from some of them (@dylanb @NSoiffer @fred-wang) before we get rid of it completely. While maybe not the optimal solution, it's probably providing some benefit to users with screenreaders, which can understand MathML much better than the HTML we produce.

If we do want to transition to using our text-based approach, I'd strongly advocate for us always producing it (like we do with the MathML now), instead of just allowing people to include it, because most users will end up not including it which would be very sad for accessibility. Similarly, I'd argue against allowing turning the MathML off, it's there because people need it.

[dylanb]

I think it would be good to separate out the two different issues:

1. Make the parse tree better (documented, more regular etc.), and
2. Should the output continue to support MathML?

It seems to me (and I am just a casual observer), that you are using KaTeX for the equivalent of presentational MathML and you are using MathML for semantic markup.

Am I correct in assuming that the ideal scenario for you would be if you could rely on MathML for the semantic presentation and no longer have to develop your own serialization?

If this is the case, then the answer to whether you should drop MathML support should probably derive from the answer to the question of would it be easier (less work) to re-implement MathML in X (pick a time frame) years (if it has become mature) or would it be easier to continue to support both?

Additionally, you might want to consider the issue of languages - how many languages will you need/want to support and when. MathML, being semantic, is language independent and will translate automatically on any system that supports MathML (assuming this system also supports the required target language).

My gut instinct is telling me that the right thing is to both continue to improve your technology, while at the same time maintaining MathML support.

[NSoiffer]

With MathML in the page, people who use the leading screen readers (NVDA & JAWS on Windows, VoiceOver on OS X/iOS) can listen to the math, navigate the math to understand it better, or read it in braille (there are special math codes for braille). TextHELP, a leading learning disability tool, uses it also. So does Chrome's ChromeVox screen reader.

@kevinbarabash said "...for the browser/screen reader combinations that do support it, what is read out isn't actually very helpful." While there is lots of room for improvement in what is spoken (MathPlayer has over 1100 rules to generate more natural speech, and it is far from complete -- see the Clear Speak documentation to get an idea of why there are many cases), saying that what is there isn't very helpful is simply not true. Similarly, citing one Wikipedia author ranting about MathML display is a poor excuse as justification for dropping MathML, especially when you don't even use it for display!

The speech needed by different disability populations differs, so a one size fits all "alt text" solution is bad. Further more, alt text doesn't allow for a braille translation or exploration/navigation of the math. Nor does it allow for prosody (mainly pauses) to make the math speech understandable. In English, there is the further problem that most speech engines speak an "a" with the short "ah" sound when reading math so even something simple like "a x = b" is hard to understand.

Alt text has the additional problem that the speech for the math needs to be localized to the different languages supported by Khan Academy. Both VoiceOver and NVDA+MathPlayer support multiple language for speech. From personal experience, getting translators who are knowledgeable about math terms is not an easy path to follow.

An alternative to using "text" alt text is to use TeX as the alt text. I'm not clear if someone proposed that or not. In case someone thinks that's a good idea, Wikipedia use to deliver TeX as alt text for their images of math. Most people in the accessibility community considered it useless. Type some TeX into a some text editor and listen to a speech engine speak it. Even if you are familiar with TeX, it's very hard to understand... and most accessibility users aren't familiar with TeX. Also, as with "text" alt text, you aren't getting math braille and you can't navigate it.

Please don't take a huge step backwards in accessibility of your site. Reach out the accessibility community to understand their needs before you cut them off at the knees (perhaps not the most PC turn of phrase, but I think very apt).

[dylanb]

Also, you might find this interesting http://accessibility.pearson.com/mathex-app/index.html

[gagern]

@dylanb wrote:

It seems to me (and I am just a casual observer), that you are using KaTeX for the equivalent of presentational MathML and you are using MathML for semantic markup.

We are currently using KaTeX to turn LaTeX into two forms of output: HTML for a visual presentation and an invisible element of presentational MathML in the hope that it helps with accessibility. We don't have semantic MathML anywhere, and as turning presentational markup into semantic markup is impossible except in the simplest of cases or with sufficient additional information, we don't expect to get that. This also holds for turning the formula into a semantic verbal description of the formula, so we probably would either end up with errors or with the verbal analogon of a presentational description.

@NSoiffer wrote:

An alternative to using "text" alt text is to use TeX as the alt text. […] Most people in the accessibility community considered it useless. Type some TeX into a some text editor and listen to a speech engine speak it.

Does the same hold for braille displays? In my very limited experience (based on interaction with exactly one person), LaTeX on braille was the preferred mode of conveying math formulas. And to me it does make a lot of sense, since it's far more concise than the spelled-out verbal formulation. Personally I'd prefer LaTeX over a verbal description when reading, but not when listening.

Of course, it might be even better if we could turn our input into Nemeth Braille (mentioned by the Murray Sargent posts @ronkok referenced), but I believe that won't work for all inputs. I think it is possible that for those formulas where Nemeth notation doesn't work (as they uses exotic symbols or constructs), LaTeX on (8-dot) Braille might be better than LateX to MathML to whatever on Braille.

[edemaine]

It's maybe worth mentioning that KaTeX also produces a hidden <annotation encoding="application/x-tex">...</annotation> within the MathML that includes the LaTeX source. Not sure whether this helps with screen reading, but in any case we shouldn't remove it. (Would be great for copy-paste support, for example...) Also, do screen readers deal well with presentation MathML (as currently generated), or do they need semantic MathML?

Seems like one of us should give some screen readers a test drive to see how this all works. I take it a reasonable combo is NVDA and MathPlayer?

[sinabahram]

Hi all, I hope everyone is doing well. I want to start by thanking you for the strides in accessibility you have made so far. For blind computer scientists like me, it is so often the case that ignorance, not bad intentions of any variety, is the source of so much frustration when accessing the world’s digital information.

By including MathML, you allow me to hear the mathematics using a variety of screen readers on various platforms. You allow me to interact with the mathematics to semantically zoom in, zoom out, navigate across different granularities at a myriad of verbosities, read the mathematics in nemeth code (a Braille code used for the display of mathematical information), and so much more.

I come to this thread with some expertise and context. I am fortunate enough to be one of the individuals who has contributed to this body of work through a variety of projects and collaborations, not the least of which is multiple years spent co-designing the interaction models behind MathPlayer 4. My background is in computer science, having undergraduate and graduate degrees in the field. I now run an accessibility consulting firm that focuses on collaborating with organizations of various sizes around the world on inclusive design projects.

It is difficult to overstate the delta of experience, usability, and accessibility between MathML being available and its absence. I urge you to please leave this critical and impactful work within the code base. I understand, am sympathetic to, and am completely willing to discuss ways to minimize the code surface area impact effect of leaving MathML there, making it more usable to more folks, allowing for different presentation formats to be used, etc., but I urge you in the strongest possible terms to please not make your service inaccessible to those users who have the actionable opportunity that I did not have in my education, which is to have accessible and significant access to digital mathematics.

I thank you all again for the time and effort you put into this and other technical discussions. It is so often overlooked by those who simplify such tasks into a simple coding problem.

[kohler]

The important comparison is between MathML as interpreted by screenreaders and Khan Academy's text translation output, and it would be wonderful to have a test page that allowed users to make that comparison themselves.

[NSoiffer]

Although @sinabahram answered many questions, I want to make sure everyone is clear on what screen readers do with presentation MathML:

1. They produce speech;
2. They produce Nemeth code and potentially other braille math codes (there are several used around the world) on an attached refreshable braille display;
3. They allow navigation/exploration of the expression.

If the MathML is rendered, TextHELP's products will highlight the subexpressions as they are spoken. See this example if you aren't clear about what I mean. By putting the MathML in the page, even if it is hidden, script can be written that unhides it and deletes the "HTML math" (made easy when you consistently mark it with a class like "katex-html"). I don't think TextHELP does that now, but with the MathML there, it's possible.

The Pearson editor/app mentioned by @dylanb is the only editor that I know that is based on content (semantic) MathML. It generates presentation MathML for display. They support both braille input and output. It's meant to be used in testing applications and the content MathML makes it easier (but not easy) to determine if the answer to the question is correct since content MathML abstracts away potentially different notations used. There are still many equivalent expressions it must worry about, but at least some parts of the equivalency problem go away.

@kohler: if you want test pages with examples of speech output, there are thousands of examples of ClearSpeak's output in the ClearSpeak spec and test cases. I'm biased since I was the primary author for MathPlayer, but I think NVDA+MathPlayer will give you the best speaking experience (and it's free). MathPlayer gives you the choice of three different speaking styles, with MathSpeak and SimpleSpeech being the other two. It also has controls to target different user audiences along with subject-specific speech rules. GH Braille hosts the MathSpeak spec ; there are lots of examples of MathSpeak output there. I suggest grabbing examples from those pages (you can get TeX or MathML for the ClearSpeak spec & tests if you have MathType [free 30 day trial]), so it should be easy for you to copy them to other formats for testing the Khan Academy's text translation output.

[kevinbarabash]

I think it's safe to say the result of this discussion has been that we will continue to support MathML rendering to support people using screen readers. Recently, we switched our test runner to jest so that we can use snapshot testing to make it easier to test our MathML output. I fixed a couple of small issues with our output. Currently the largest open issue that we should be collating numbers, see #203.

[hungrymonkey]

https://bugs.chromium.org/p/chromium/issues/detail?id=6606
https://mathml.igalia.com/news/2020/02/19/upstreaming-progress/#new
Igalia started upstreaming MathML into Chromium.

[NSoiffer]

You can follow the MathML progress on the community group page. In particular, the current draft of MathML Core (the part of MathML that will be implemented in browsers -- the rest will be handled by a polyfill/library) is linked from that page.

Anyone is welcome to join the CG. See the CG page for links to join.

[kevinbarabash]

According to https://mathml-refresh.github.io/mathml-core/#css-styling it looks like the standard will support custom fonts for MathML rendering. I wonder if anyone has package Computer Modern fonts as OpenType fonts with an appropriate MATH table. That would in theory allow people to render MathML and have it look relatively close to KaTeX/LaTeX rendering.

[fred-wang]

@kevinbarabash See http://www.gust.org.pl/projects/e-foundry/lm-math