Expose MathML for accessibility

[jcsteh]

In order for screen readers and other assistive technology (AT) products to provide rich access to math content, they need to be able to access the raw math in a standard way. Providing alternate text is not sufficient, since although this does allow for better spoken math, it does not provide for braille, interactive exploration, etc. It seems MathML has become the dominant standard for exposing math content and MathJax uses MathML internally, so it'd make sense to expose it as MathML.

You can already use the toMathML extension to access MathML for a MathJax tree. The problem is that this requires JavaScript calls and an AT does not always have the ability to execute arbitrary JavaScript on the page. (For example, in Firefox, this isn't possible without a browser extension or Greasemonkey script.) There needs to be some way to obtain MathML from MathJax that does not require JavaScript calls.

One solution is to expose the MathML via an attribute; e.g. data-mathml. This is pretty inelegant and non-standard, but it does solve the problem. I've already implemented experimental support for this in NVDA's (not yet released) math support.

If exposing an attribute like this automatically is considered too expensive, we might be able to come up with a solution using an off-screen button or similar which exposes the attribute when activated. However, this is even less elegant and is more error prone, so I'd hope to avoid something like this if possible.

/cc @pkra

[jcsteh]

/cc @NSoiffer @camlorn

[jcsteh]

I created a Greasemonkey script to do this for MathJax on any page with the toMathML extension loaded. Unfortunately, this isn't working with latest Firefox + Scriptish because of security changes in Firefox. I'm not sure about Greasemonkey.

[pkra]

Thanks for creating the bug! I'm guessing this can start out as an extension and assuming no hidden issues show up, it could ship with the combined configuration files (which most people use).

As for the implementation, an easy (though somewhat strange) way would be to simply attach the data attribute it to the (internal) MathML -- thanks to #860 this will be preserved. Alternatively, the solution for #860 could be re-used.

[pkra]

Related to #939, I guess another option should be to trigger this in via an (accessible) menu so that even in cases where someone custom build a configuration the user can trigger this.

[ahicks92]

At the moment, per-site, it's possible to turn it on. It takes something like 10 keystrokes and rather esoteric knowledge from the perspective of most users. I.e. of all the blind people I know, there's like 5 who would actually be able to follow my instructions, and one of them is @jcsteh. Definitely +1 to this.

[jcsteh]

I neglected to mention that when this exposure is enabled, the root of the MathJax output tree (where this attribute is exposed) should also expose the attribute role="math".

[pkra]

Thanks for the additional comments!

[NSoiffer]

Both Wikipedia and Kahn Academy expose MathML. For the record, here's what they do:

Wikipedia

After setting the math display option to MathML with SVG or PNG fallback

FireFox

class="mwe-math-mathml-inline mwe-math-mathml-a11y" style="display: none;"

.mwe-math-mathml-inline {
    display: inline !important;
}

.mwe-math-mathml-a11y {
    clip: auto;
    overflow: visible;
    position: static;
    width: auto;
    height: auto;
    opacity: 1;
}

Chrome

.mwe-math-mathml-inline {
    display: inline !important;
}

.mwe-math-mathml-a11y {
    clip: rect(1px,1px,1px,1px);
    overflow: hidden;
    position: absolute;
    width: 1px;
    height: 1px;
    opacity: 0;
}

Kahn Academy

The HTML-CSS ugliness that KaTeX generates has the aria-hidden attr on it. The MathML (class="katex-mathml") makes itself small so it isn't visible:

.katex .katex-mathml {
    position: absolute !important;
    clip: rect(1px, 1px, 1px, 1px);
    padding: 0px !important;
    border: 0px none !important;
    height: 1px !important;
    width: 1px !important;
    overflow: hidden;
}

[pkra]

Thanks so much for those details, Neil. As you can see, this issue is in the milestone for the next release.

However, the OP proposes a different solution -- exposing MathML in a data-attribute so that AT can pick it up more easily. My impression is that, at this point, embedding MathML in the DOM is really only necessary for VoiceOver (and at least Wikipedia has a bug report for VoiceOver not working with its embedding). IIRC, all other math-enabled screenreaders can pick up MathJax's internal MathML already, so we're hoping that covering these two use cases will be a good iteration.

In any case, we'll bring our proposal up for discussion here and on the dev-list and reach out to other AT vendors.

[NSoiffer]

I think the only AT that can pick up MJ's internal MathML is JAWS in IE
only. IE is the only browser that makes calling JS not too difficult. If
you embed the MathML in the DOM as opposed to "hiding" it or not exposing,
then I think AT will "just work". E.g, when Kahn Academy did it, NVDA
started working with it without any new work.

AFAIK, I've convinced the JAWS folks to look for on a page because
trying to get it to work in Firefox and Chrome was a lot of work and
fragile. Having to do something special based on what technology is
rendering math isn't a great solution, so doing something that works
independent of MathJax running or not would be best as it removes the need
to do special casing.

The one place where maybe you need to know about the renderer is for sync
highlighting. But if AT could just tweak foreground and background of the
mathml in the DOM and the renderer would pick that up, that might be best.
That's a distant future though probably based on ShadowDOM. On the other
hand, the interface we worked out with TextHELP is probably a lot easier to
use for AT.

[sinabahram]

This is true.

In fact, I wrote a bookmarklet, MathInjector, to extract data-mathml and stuff it in a page with MathJax.

https://github.com/SinaBahram/MathInjector

I’d love to avoid ever having to use said bookmarklet e.g. AT does it automatically. There are some encoding issues, though (see comments in bookmarklet).

[jcsteh]

data-mathml is a massive hack. The only reason I suggested it was to avoid exposing real MathML in the DOM and having to visually hide it somehow, as well as having to hide the HTML-CSS rendering for accessibility. Real MathML would of course be much nicer, but see below.

There are two problems with these "hiding" solutions:

1. Because there's no explicit facility for having content visible to accessibility but hidden visually, different ATs may handle it differently.
2. Even if you find an approach that works for all ATs, ATs very possibly won't get the on-screen location of the math being visually displayed. For example, taking what Kahn Academy does, the AT will be told that the math has a width and height of 1 and the left and top coordinates might even be wrong too. This means, for example, that AT users won't be able to invoke the MathJax context menu. If the AT wants to display its own representation of the math for some reason, it won't know where to position it. I'm not sure about this one, but it may cause problems for magnifiers too, since they may not be able to get the location information they need.

I seem to recall hearing (but could well be wrong) that JAWS can only pick up MathML from MathJax if the input was MathML (and thus the MathML is hidden in the DOM); i.e. it won't work if the input was LaTeX. If that's the case, they're going to need a solution like this too to support other input formats.

[sinabahram]

I’m typing out loud here.

Regarding the first point, while it’s not an explicit thing, we’ve used off screen content for over two decades. I think there’s a nice combination of CSS that works across Jaws, NVDA, VoiceOver, within Firefox/IE/Chrome/Safari, etc.

I’ve not seen any AT not pick up on such things. At least, no AT that I’ve wanted to have access to it e.g. I obviously wouldn’t expect zoom text to pick up on it, and that’s a good thing, since it’s being hidden visually.

Regarding your second point, wouldn’t a reverse aria-describedby be a good idea here? we can put an aria-describedby on the image or other silly representation being chosen, and assign it’s value to the id of the off screen content. I only say silly because I think we’d all prefer that the MathML be the true representation, but unfortunately, given the lack of rigger in its visual presentation, certain mathematicians, I’m told, still prefer what they can achieve via LaTeX typesetting and other means.

There are accessible name calculations to think through, but if the accessible name of the mathematics is simply the output from running it through mathplayer, and or that can be assigned to it at runtime via an injection of aria-label onto said MathML, then the describedby pointing to it should pick that up and slap it onto the element being displayed visually. Since we’re doing an aria-describedby instead of an aria-labeledby, it won’t matter if MathPlayer isn’t installed, thus returning nothing from the aria-label of said MathML. E.g. graceful fallback to alt of the image.

Thoughts?

[pkra]

Thanks for all your comments -- we really appreciate it.

Re @NSoiffer

I think the only AT that can pick up MJ's internal MathML is JAWS in IE only.

IIRC, ChromeVox, Texthelp and GH all use MathJax directly in various settings (web and apps), including synchronized highlighting.

If you embed the MathML in the DOM as opposed to "hiding" it or not exposing, then I think AT will "just work".

From what I've heard, all hidden embeddings so far have had problems, so I'm a bit sceptical (also when ignoring the performance penalty for this).

Thanks also @jcsteh and @sinabahram for those valuable points. @sinabahram, aria-describedby is on our todo list to investigate; do you happen to know if there's a way to specify multiple alternatives? We can imagine a situation where we would provide MathML, TeX and speech text possibly more author-provided formats. (Also, we would want to expose the MathML tree structure in the output.)

The larger problem is that native brower support is not on the horizon and web components are also not very close to being usable in production (neither natively nor with polyfills, which IIUC also cause a11y problems).

So we think it would be a good idea to develop a practical and generic way of exposing the underlying data (i.e., MathML but potentially other source) in the rendering. In my eyes, this would ideally be something we could publish as a note of the MathWG.

The goal would be that all MathML renderers and AT solutions could adopt it, independently of any one rendering solution or any one AT math-component. It could even help accesibility solutions for other formats move forward (e.g., ChemML).

This does not seem unattainable to me.

Perhaps it makes sense to set up a call with interested folks as soon as possible. For those interested, drop me an email at peter.krautzberger@mathjax.org.

[jcsteh]

@sinabahram: While you may be correct about off-screen content in most cases, @pkra noted that there's a bug against Wikipedia regarding VoiceOver not detecting the MathML. Even if this is somehow incorrect, the point is that this is highly unreliable/error prone.

IMO, aria-describedby isn't an option here for several reasons:

1. An AT would still need to have explicit support to hunt for the node referenced by aria-describedby and handle it as math. It'd also need to map the on-screen location to the describedby node. If we're going to do that, we may as well just do data-mathml, which is less convoluted (if still hacky).
2. Even if an AT did this, the math would appear twice to the user, since the description (as referenced by aria-describedby) would still be present in its original place as well. (We don't "move" the description from its original place to the place where it is referenced for various reasons.)
3. Browsers/existing ATs would try to calculate text from the math referenced by aria-describedby, resulting in a very confusing description.
4. The output from MathPlayer can't simply be made the accessible name. It contains stuff other than text (pauses, etc.), which accessible name can't handle.

Basically, using aria-describedby is still a nasty hack, is more convoluted and doesn't solve any problems better than data-mathml. I'm very much opposed to this solution. :)

[sinabahram]

Great points, and thanks for making them.

So, it seems like data-mathml is the way to go, barring any other suggestions?

Should we agree on a standard encoding?

[pkra]

Here are some notes coming out of our last developer meeting. I'll send out an email to schedule a call. Not sure how to reconcile the time zones involved but we'll sort it out.

Some background.

• MathJax attaches ids to all elements of its output (for various reasons)
  • if the source includes an id (e.g., a MathML element), this id will be re-used for the corresponding element in the output.
• The new CommonHTML output can be highlighted directly, a highlighting extension for the HTML-CSS output is in the works.

Proposal(s)

• Structure of the exposed MathML
  • The exposed MathML will correspond to the internal MathML used to generate the output (not necessarily identical to the source)
  • Each MathML element will carry an id ( if preferred: a data-* attribute) corresponding to the id of the element corresponding to it in the output.
  • This allows AT to easily link the MathML structure with the output structure.
• Options for embedding
  1. Add a data-mathml attribute with serialization of the MathML (with ids)
     • mapping to output
     • AT will serialize anyway (true?)
     • simpler implementation, better performance
  2. Add hidden MathML directly (without ids)
     • MathML is in the DOM
     • accessible to DOM exploration
     • no mapping to rendering
     • rendering needs to be hidden (or aria-describedby used but as discussed this has other problems)
     • could be made to work with VoicOver
  3. Add hidden <object> element containing the MathML (with ids).
     • MathML in the DOM
     • accessible to DOM exploration
     • retaining a direct mapping via the ids (since <object> elements are a separate document there is no conflict).

The team is leaning towards (i.) but we want to implement what works best for AT (or more pricesly, its users).

[jcsteh]

Thanks for the great summary. Regarding serialisation, NVDA does serialise, but it's possible that VoiceOver doesn't serialise into MathML, as it obtains math via the accessibility hierarchy. That may preclude data-mathml as an option if we want VoiceOver support. I guess someone will need to discuss this with Apple accessibility. It's worth noting that visually hiding the MathML and hiding the HTML-CSS for accessibility would certainly work for both NVDA and VoiceOver right now. However, it does have the disadvantages I noted regarding associating to the visual location, etc. Primarily, this means no access to the MathJax context menu and no ability to display an overlay over the math. The former in particular is pretty serious.

[pkra]

Thanks, @jcsteh.

[...] That may preclude data-mathml as an option if we want VoiceOver support.

Right, we are not assuming data-mathml would work with VoiceOver; a specialized solution would probably be necessary (just like it is for MathPlayer 3).

I guess someone will need to discuss this with Apple accessibility.

We don't have any contacts there so we'd appreciate any help.

It's worth noting that visually hiding the MathML and hiding the HTML-CSS for accessibility would certainly work for both NVDA and VoiceOver right now.

Right. We're open to any solution that helps AT users the most.

[pkra]

[Ah, I somehow triggered a posting. Moving the second comment into the right place]

[NSoiffer]

I came across this Paciello Group post that includes tests on browsers and AT (updated Nov 2013). It recommends the off screen technique:

Use an off screen technique to remove content from visible display, but still available to screen reader users:
Example code:
.offscreen
{position:absolute;
left:-10000px;
top:auto;
width:1px;
height:1px;
overflow:hidden;}

<div class="offscreen">This text is hidden.

They said this recommendation comes from a WebAIM page on hiding in CSS

Absolutely positioning content off-screen

Using CSS to move hidden elements to a position off-screen is generally accepted as the most useful and accessible method of hiding content visually.

It seems to me we should go with the established techniques and then try to mitigate the menu issue that James raised if it really is an issue (the menu is currently not accessible AFAIK). Mitigating the overlay issue (if it is one) could potentially be done by having whomever cared look at the next element (of a certain kind?) as that would be the display.

Of course, none of this is needed with Shadow DOM (I think)...

[sinabahram]

Please see this page for a few enhancements to that technique:

http://snook.ca/archives/html_and_css/hiding-content-for-accessibility

[fleizach]

Hiding MathML in an attribute would make touch exploration of math elements impossible with VoiceOver on iOS.... ditto for moving it off screen

The best option is to use MathML natively. Then it's there visually, it can be explored by touch and the native rendering semantics that are deployed when rendering in web engines is able to be leveraged by accessibility code to output relevant information.

[pkra]

Just an update for people on this thread. We've finished work on #939 (making the MathJax menu accessible).

I've updated the example mentioned earlier. We'd appreciate any and all feedback!

[pkra]

Ack -- @dpvc I just realized that the AssistiveMML branch is not yet merged into develop. So the updated demo is missing the extension. When do you think that will be merged?

[dpvc]

I want to fix the aria stuff first, since the extension currently has to manipulate those (which was just intended as a temporary solution). I hope to be able to work on it this weekend.

[pkra]

@dpvc no problem. I've reverted my change to the AssitiveMML demo (it might take a while until rawgit picks up on the change -- sorry about that).

I've created a separate demo for the menu at https://rawgit.com/pkra/4c57afc70c4f1a4fb356/raw/450265ccd8ed8ccfc78d2ad63fe988f96c4bc915/mathjax-accessiblemenu.html

[sinabahram]

That sample page does not work with any combination I tried. The four I tried quickly were NVDA+Firefox, Jaws16+Firefox, NVDA+IE11, and Jaws16+IE11.

I wasn’t able to read any of the mathematics as it did not show up as mathml on the page.

[fsrg]

FYI: JAWS will only use MathJax in IE.

Thanks,
RG

[pkra]

Thanks for the feedback!

That sample page does not work with any combination I tried.

This is totally my fault. Since I now mentioned two (three, four) different pages in the thread I can't tell which page you might refer to. My fault -- sorry about that!

Trying to recap:

• To try the Assistive-MML extension (with embedded MathML for screenreaders) go to https://rawgit.com/pkra/ac7d48279728d3ab5881/raw/adfb992266802e7fa5a3562a4f96d33b229fc853/mathjax-assistivemml.html
• To try the accessible menu (equation with tabindex, trigger menu with space etc) go to https://rawgit.com/pkra/ac7d48279728d3ab5881/raw/adfb992266802e7fa5a3562a4f96d33b229fc853/mathjax-assistivemml.html

Thanks for any additional feedback!

[sinabahram]

Ok, three out of four:

NVDA+Firefox works

Jaws+IE and Jaws+firefox work

However, NVDA+IE does not.

At a client site all week, but as I grab few moments here and there, I can test the menu and anything else you throw on here. thank you for all of your work on this!

[sinabahram]

No, Jaws works fine with firefox as of a few updates ago. I can provide links if that’s helpful.

[pkra]

Re @fsrg

FYI: JAWS will only use MathJax in IE.

Actually, we had good results with JAWS on Firefox using the new extension, cf. mathjax/MathJax-docs#111

Re @sinabahram

However, NVDA+IE does not.

That's #1235 and seems to be caused by NVDA.

@sinabahram does the menu work for you (to some degree)?

[fsrg]

JAWS does support MathML in Firefox. But it does not access content via MathJax in Firefox.

Thanks,
RG

[pkra]

JAWS does support MathML in Firefox. But it does not access content via MathJax in Firefox.

Well, it does work anyway with the AssistiveMML extension :-)

[dpvc]

FYI: JAWS will only use MathJax in IE.

This sounds like JAWS is detecting MathJax in IE and calling on it directly? In that case, will the added MathML in the DOM cause problems (like reading the math twice) for JAWS in IE?

[fsrg]

I don’t think so, provided that the MathML in the dom is inside the MathJax element JAWS looks for when determining if it has encountered MathJax. As I recall, it is one of the top level elements.

Thanks,
RG

[pkra]

According to feedback from other JAWS users and my own testing, JAWS does not read things twice in IE. See the MathJax-docs issue for further details.

[dpvc]

OK, great! That sounds good (the MathML is inside the top-level MathJax element container).

[dpvc]

I've merged the assistive-mml branch into develop.

[sinabahram]

If there’s a new test page, I’m happy to test the standard four configurations.

And, I’m happy to also test the menu and tabbability of the math.

[dpvc]

I'm closing this issue, since the code is merged, and we are in the testing phase for the v2.6 release. If any new problems are discovered, a new issue should be opened for them at this point.