Make KaTeX accessible

[dylanb]

Make KaTeX accessible to people with disabilities in particular screen reader users as they are most affected by the current limitations

[xymostech]

Hey! We've been thinking about ways to make KaTeX accessible, and we weren't sure what would be the best way. Is something like adding a tag to the element that lists the TeX expression good enough? Do we need to render to MathML also? I'm not very experienced with this (the other contributor @spicyj is more so), so any advice would be great!

[dylanb]

MathML is definitely the most "future-proof" solution but unfortunately does not work everywhere yet so the answer depends to a large degree on your platform support requirements/desires. Am I correct in assuming that you want to support all platforms? How far back do your for example want to go in supporting IE?

[xymostech]

We go back to IE 8 in our support. We're not planning on supporting any further back. We'll talk about what it would take to support MathML. Would this be something that you would enable only when someone is using a screen reader?

[ctoth]

Absolutely not. Screen reader detection is not reliable, and there are
other potential use cases/technologies that render MathML outside the
traditional screen reading paradigm.

On 9/15/2014 2:46 PM, Emily Eisenberg wrote:

We go back to IE 8 in our support. We're not planning on supporting
any further back. We'll talk about what it would take to support
MathML. Would this be something that you would enable only when
someone is using a screen reader?

—
Reply to this email directly or view it on GitHub
#38 (comment).

    -Q

Accessible Software for the Blind

http://GetAccessibleApps.com

[fred-wang]

More generally, it would be good to expose a MathML output so that users of accessibility tools or of browsers with native MathML support could switch to that mode instead.

[xymostech]

Unfortunately, rendering to MathML all the time is impractical because Chrome seems insistent on not supporting it: https://code.google.com/p/chromium/issues/detail?id=152430 :(

[xymostech]

Yes, we'll definitely look into supporting it as an alternative output. I'm just curious how people with screen readers would enable it.

[fred-wang]

If you expose both MathML and your HTML output, then it would be possible to write browser extensions to automatically switch to MathML for people who want that (e.g. https://addons.mozilla.org/en-US/firefox/addon/mathjax-native-mathml/ or https://github.com/fred-wang/mathjax-native-mathml-safari). You could also write some kind of UI like MathJax does (it uses a menu and a preference cookie).

[xymostech]

A browser extension definitely sounds reasonable for exposing that functionality.

We're fairly opposed to a MathJax-like UI, because it's annoying and often is not helpful.

Another thing to think about is that we currently support a way to render on the server and then serve up plain HTML, with no JS required. Maybe we could render to both on the server, serve up both, and switch between them (with a smaller bit of JS) depending on what's needed? Or we could serve up the HTML and the JS, and then switch to the MathML if you have the extension...

[fred-wang]

For server-side, I think the best is to served both markup (MathML and HTML) so that math search, feed reader and other web crawlers could have access to the MathML (cc'ing @brucemiller, @physikerwelt, @kohlhase, @dginev, @huxuan who have done some work on math search). The overhead of the MathML conversion/source should be small compared to the HTML one.

[dginev]

Oh, lovely, we now have a serious MathJaX-like competitor. While I am extremely happy to see there is effort invested in better math on the web, it also pains me to see projects with a big component dedicated to circumventing the stubbornness of mainstream browsers. I went back and bumped the Chromium issue, has been a while since I talked to that wall:
https://code.google.com/p/chromium/issues/detail?id=152430#c99

I heartily +1 the suggestion to have a MathML output, as it is the standard markup language for representing mathematics in XML documents, as well as on the web (HTML5). ePub3 is another MathML-enabled format, where this could be really handy.

[MoritzGiessmann]

+1 for MathML support instead of span elements!

[dylanb]

If you want broad support including IE8, then your only solutions if you do not always want to render MathJAX are:

0. Implement MathML and your own rendering by default, use aria-hidden to hide your rendering from screen readers.
1. Allow the user to turn on MathJAX (for support on platforms that do not support MathML) - a11y people will tell you that this is self identification by people with disabilities, and balk at an approach that only does this,
2. Hybrid of 0 and 1, where instead of MathML, the user can switch to MathJAX on platforms that do not support MathML
3. Implement a natural language renderer of math

Independent problems:

a) When you have a combination of input fields within the math (for example in an assessment), where do you put these input fields? MathML does not support this. A natural language render could support this. Annotations within MathML (for the MathML approach) have been suggested as a way to deal with this but support for this is untested as far as i know.

If you have the resources, I would suggest #3 because if you can make this code as fast as your current code, then you would have made the world of accessible math much better. Biggest problem here is translation.

[xymostech]

What do you mean by "Implement a natural language renderer of math"? Does this mean turning "x^2" into "x squared"? I think that the easiest solution for now will be to implement MathML. In browsers that don't support MathML (like chrome), is the MathML still accessibile even though it isn't rendered correctly, or does the browser need to have support for it in order for screen readers, etc to understand it?

Thanks for helping think through this, everyone!

[kohlhase]

You should really invest in MathML (content & presentation in parallel markup). You should see MathML as a standardized interface that allows externals tools to do what they do best. A11y is a good example of this, if you have MathML, you can feed it to a Math-aware screen reader. There are multiple examples of this, e.g. formula reader by Design Science that is integrated into Mathplayer. With MathML, a11y tools share a common basis and can compete on merits. A similar case is Math Search, there are various search engines that take MathML (content/presentation) as input and offer different functionalities on this basis (see e.g. http://ntcir-math.nii.ac.jp/ for a Math IR challenge and an overview).
In the long run, everyone wins from having open standards.

[fred-wang]

In browsers that don't support MathML (like chrome), is the MathML still accessibile even though it isn't rendered correctly,

Yes, with chromevox. I think Design Science is also working on a library that works on various browsers, including Firefox and Chrome.

or does the browser need to have support for it in order for screen readers, etc to understand it?

An additional note: for WebKit Mac, the Accessible tree relies on information from the render tree (so for example this does not work very well if you hide the math with "display: none", even if you use some aria attributes to make the MathML important for accessibility...).

[sophiebits]

I was imagining we would mark the MathML with CSS along the following lines:

  clip: rect(1px, 1px, 1px, 1px);
  height: 1px;
  overflow: hidden;
  position: absolute;
  width: 1px

and mark the presentational HTML with aria-hidden.

[fred-wang]

This will certainly work for DOM-based accessibility tools. That might work for Safari+VoiceOver but I can't tell for sure. As I recall, Chris Fleizach (a10y engineer at Apple) suggested something along these lines when we discussed that for MediaWiki.

[sophiebits]

I believe I tested a similar setup with VoiceOver a couple months ago and it worked. It's what Facebook uses for some accessibility hints in their interface.

[dylanb]

@xymostech The screen reader support varies and requires additional plugins in order to work on all platforms (MathPlayer is required on Windows machines for example). The bottom line is that you will have to test support on all your platforms. If additional plugins are an acceptable solution for your users, then MathML might be the best approach.

@spicyj The clip rect mechanism is one of the offscreen mechanisms that work well with all screen readers because the browser still exposes the HTML structure in the accessibility tree.

[NSoiffer]

+111111 for using MathML.

It is a rather strange situation that accessibility support for MathML is ahead of mainstream rendering support. In Safari, Apple has added accessibility support (speech and navigation) for MathML in connection with VoiceOver. The visual rendering in WebKit is still weak, but it is getting better thanks to Fred Wang and others.

In Chrome, ChromeVox can speak and navigate the math even though it is unable to render it. So if you hit a page with MathML, you can turn on ChomeVox to listen to and understand what the browser can't render :-) Of course, most people use MathJax to deal with this problem.

IE8 & IE9 have MathPlayer to do the rendering and the speech and most AT software works with it. Sadly, IE stopped support for plug-ins, so MathPlayer doesn't work in the latest versions of IE. However, as someone mentioned, we are (hopefully) close to releasing a replacement that will speak and navigate math in all browsers. With MathJax rendering MathML, you have an accessible solution. Also, IE11 added an "enterprise mode" partly to support legacy plugins. Unfortunately, it's buggy and IE crashes when MathPlayer renders math in some cases. There's an open bug on this and maybe it will get fixed. If so, there's a path to having visual display and accessibility in IE with MathML.

With that background, you can see why so many people suggest using MathML for accessibility.

The problem that you face, and one that MathJax also faces is what to do when rendering non-MathML input. I see three short term solutions:

1. Expose the MathML based on some setting. As mentioned, you don't want to do this all the time due to problems with rendering it in all browsers.
2. Do what MathJax does and expose a function that returns the MathML. Unfortunately, that's not a good solution for screen readers because external (C++, etc) programs can't universally call JS. You can call JS in IE, and I think you can in Firefox if you write an add-on and the C++ calls the add-on via XPCOM, but I haven't found a way to do it in Chrome [if anyone knows, please provide a link].
3. Use a hack where you put the MathML somewhere it won't rendering such as in a script tag or in a data-mathml attribute on some span or other element. That will only work if you can convince Apple and ChromeVox and AT vendors to look for the MathML wherever you hid it. Khan Academy is a pretty important website, but I don't know if it is important enough to get Apple and others to do something special in their accessibility solution.

The better long term solution is to consider using a shadow DOM assuming it catches on. That way you expose MathML to the world but stick your (ugly) spans in the shadow DOM and no one has to see your implementation. Looking at http://caniuse.com/#feat=shadowdom, Shadow DOM only currently works with Chrome and with Firefox with the dom.webcomponents.enabled flag. There is a polyfill for it though.

There is one more associated (maybe hard) problem you should keep in mind: good accessibility of math allows for synchronized highlighting of the math and speech. It's what almost all learning disability tools do with text and what they do when connected to MathPlayer. To do this, KaTeX needs to have a method to map between the MathML and the rendering of that element. I won't extend this lengthy comment even further with details on that, but I would be happy to discuss them when you get to thinking about the problem.

I do hope you make accessibility a priority. It's not right that an important learning site such as Khan Academy is at best only partially usable by persons with visual disabilities -- it should work for all people who want to or need to use its great lessons.

[Hywan]

+1 for MathML :-).

[ricogallo]

Agreed 👍

[mathiasbynens]

Since web components were mentioned in this thread, I’ll just leave this here. I’ve made a small vanilla web component wrapper around KaTeX: https://github.com/mathiasbynens/math-tex

Demo: https://mathiasbynens.github.io/math-tex/dist/example.html

[dylanb]

@mathiasbynens what problem was this supposed to solve?

[mathiasbynens]

@dylanb It makes it slightly easier to implement KaTeX. It is also a demonstration of using the Shadow DOM to hide the “ugly <span>s”.

But any accessibility problems (which is what this issue is about) need to be fixed in KaTeX itself.

[ahicks92]

Your only real option is MathML. There are interesting (read: hacky) solutions that I currently use-namely, blocking MathJax from rendering in the first place and reading the LaTeX. This is suboptimal at best, and it's rare to find a blind person who knows it.
If KaTeX is going to be rendering on the server only, then this is not an option. Net result: expose your laTeX somehow (aria, maybe) and hit the 10% or less of us screen reader users who can handle it and no one else. Follow this up with a MathML renderer as soon as possible for the other disabilities and the remaining 90% of blind people. Converting directly to speech will work for some disabilities, but not blindness: as soon as the equation becomes more than marginally complex, some form of navigation is needed and natural-sounding pauses become super, super important (there's a huge difference between the phrase "the square root of x plus three plus three" and "the square root of x plus three pause plus three").
The two major windows screen readers now both have at least some support. Jaws 16 beta has done some basic stuff, but only works with MathJax in IE (I'd not hold my breath while waiting on FS to add more support), and NVDA has some try builds that use Pymathspeak (and we're discussing programming something nicer, and I'm probably going to go start that next). NVDA does have support for the solution from Design Science, but that involves installing a component which isn't available to very many people. I know that the NVDA people have it, but have heard nothing about public betas as of yet. I can't speak to the Apple implementations, unfortunately-I've not tried them.
I also can't comment on mainstream support, but it's at least theoretically possible that some of the new accessibility laws will cause wider adoption of MathML among publishers and, consequently, pressure on all the browsers. It's almost the only way to make a math book accessible to someone blind. The other is getting your college to convert it to braille, but this costs upwards of $20000 per book. We'll see what happens.

[xymostech]

Hello everyone, thanks for all the feedback here. I've been distracted by other projects, but I'm going to be making a pull request soon about making KaTeX accessible, taking a hybrid MathML + aria-hidden HTML approach. I'd love to get some feedback and a hand in testing it out in different environments if anyone would like to help out.

[NSoiffer]

I'm glad to hear you are getting back to this. You probably should take a look at what wikipedia has done to deal with accessible math: if the user has selected the option "MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools)", they render the TeX on their server and then send down MathML and the fallbacks. To set the options, login, choose appearances, and scroll to the bottom for math options.
For more info, see this wikimedia page. The more sites that do things the same way, the more impetus there is for accessibility tools to handle that.

[Hywan]

\o/

[fred-wang]

@NSoiffer You can try it on http://katex.xymostech.com/

[NSoiffer]

That works with the new MathPlayer+NVDA in both Firefox and IE.

ChromeVox didn't work with it, but ChromeVox's math support has been broken for over a month and I can barely get it to read math anywhere, so that doesn't say much.

JAWS currently requires MathJax for accessibility, but it should be able to work with the site since it only uses MathJax to get the MathML. I pass along this site to them and hopefully they will add support in a not to distant update.

On Wed, Feb 25, 2015 at 11:58 AM, Frédéric Wang notifications@github.com
wrote:

@NSoiffer https://github.com/NSoiffer You can try it on
http://katex.xymostech.com/

—
Reply to this email directly or view it on GitHub
#38 (comment).

[brucemiller]

I'm not sure what "works" should mean here: on Firefox it is not using MathML, but span-soup. Ultimately, you'd want to send MathML to browsers that support it, right? It looks OK, though, although it's all quite huge, but I'm guessing from the CSS that that is intentional.

[NSoiffer]

If you use the develop tools in Firefox to look at the DOM, you'll see
MathML is in there. NVDA finds the MathML and gives it to MathPlayer to
speak.

As for who/what is doing the rendering, that I'm not sure about. For screen
reader accessibility, that doesn't matter. When you need to do sync
highlighting, potentially that might matter. E.g., if some code modifies
the MathML to change the background color on the mfrac, would the display
change correctly?

 Neil

On Wed, Feb 25, 2015 at 12:55 PM, bruce miller notifications@github.com
wrote:

I'm not sure what "works" should mean here: on Firefox it is not using
MathML, but span-soup. Ultimately, you'd want to send MathML to browsers
that support it, right? It looks OK, though, although it's all quite huge,
but I'm guessing from the CSS that that is intentional.

—
Reply to this email directly or view it on GitHub
#38 (comment).

[fred-wang]

Yes, the fact that the MathML rendering is hidden is a problem for assistive technologies that rely on the visual rendering such as sync highlighting or, as mentioned by Apple devs, touch screen support. However, the main issue is that KaTeX currently does not generate MathML at all, so Emily's work is already a big progress. This generates MathML + HTML with only CSS selectors to select which one that is visible, so assistive technologies can immediately obtain the MathML DOM and it's easy to write a browser add-on / greasemonkey script to force MathML to be rendered for more advanced assistive technologies or native browser rendering.

[xymostech]

When you need to do sync highlighting, potentially that might matter. E.g., if some code modifies the MathML to change the background color on the mfrac, would the display change correctly?

@NSoiffer The display would not update. Is that a serious problem? Are there lots of times when that would happen? I'm not sure if there's a good general way to watch for changes to the DOM to propogate them over to the HTML version.

[NSoiffer]

If you want the DOM to propagate to the rendering, Shadow DOM seems like
the way to go. But unless you use a polyfill, your probably got a couple of
years to wait for all the browsers to implement it.

On Wed, Feb 25, 2015 at 3:55 PM, Emily Eisenberg notifications@github.com
wrote:

When you need to do sync highlighting, potentially that might matter.
E.g., if some code modifies the MathML to change the background color on
the mfrac, would the display change correctly?

@NSoiffer https://github.com/NSoiffer The display would not update. Is
that a serious problem? Are there lots of times when that would happen? I'm
not sure if there's a good general way to watch for changes to the DOM to
propogate them over to the HTML version.

—
Reply to this email directly or view it on GitHub
#38 (comment).

[xymostech]

The accessibility patches were landed in the new release of KaTeX (v0.2.0). Hopefully the new, accessible version should start appearing around soon. (there isn't a way for developers to disable accessibility, so anywhere that it's used should have the new features)

I ran some performance tests on the new version versus the old release. It looks like we took a ~20% speed hit, which is a little sad but not bad for things being screen-readable now: http://jsperf.com/katex-vs-accessible-katex

I'm going to close this issue for now. If anyone runs into any serious problems, I'd be happy to re-open or make smaller issues. Thanks for all of the feedback, everyone!

[NSoiffer]

FYI: I spoke the JAWS developer for math support and he said they expect
JAWS will start reading MathML in the near future -- I forwarded the test
page on to him.

On Mon, Mar 2, 2015 at 12:06 PM, Emily Eisenberg notifications@github.com
wrote:

Closed #38 #38.

—
Reply to this email directly or view it on GitHub
#38 (comment).

[fred-wang]

Follow-up issues for people who care about the visual rendering too:
#197
#198

[fred-wang]

For the record, I've updated my mathjax-native-mathml extensions to force KaTeX to display its MathML output too:

• Mozilla add-on: https://addons.mozilla.org/en-US/firefox/addon/mathjax-native-mathml/
• Safari add-on: http://fred-wang.github.io/mathjax-native-mathml-safari/mathjax-native-mathml.safariextz
• Greasemonkey script (unfortunately that one does not work well due to issue rules in .katex-mathml should not be !important #197): https://openuserjs.org/scripts/fred.wang/MathJax_Native_MathML/