Add visual highlighting for web math navigation

[RyanMcCleary]

Link to issue number:

Part of #19191.

Summary of the issue:

When navigating math, there is currently no visual highlighting showing the focused subpart. Visual highlighting is very useful for low-vision users and can make it easier to use speech and magnification together while reading math.

Description of user facing changes:

When browse mode highlighting is enabled, the currently focused subpart is now highlighted while navigating math on the web. Similarly to how highlighting is handled for OCR results, math navigation highlighting is included in browse mode highlighting.

Description of developer facing changes:

The MathCATInteraction class now takes a source object as a constructor parameter (the source object is the root of the MathML being interacted with). A new file source\mathPres\MathCAT\navNodeMapping.py was also added. This file contains functions that are used to map from MathCAT's current navigation node to the corresponding IA2 object.

Description of development approach:

The main tricky part here was mapping MathCAT's current navigation node to the position of that MathML element on the screen. This is done by mapping element IDs to the corresponding IA2 object, from which the position information can be retrieved. One wrinkle in this process was MathCAT creates its own internal IDs for elements that don't already have IDs, so it was necessary to preprocess the MathML to create IDs for elements that don't have them.

Testing strategy:

Unit testing and manual testing in Chrome and Firefox.

Known issues with pull request:

The only issue I know of is the possibility that there might be a more general solution. If it is possible to implement this so that it works for Microsoft Word, Outlook, Acrobat, etc. as well, then that solution might be preferred. However, it seems likely to me that those cases will need to be handled separately.

Code Review Checklist:

• Documentation:
  • Change log entry
  • User Documentation
  • Developer / Technical Documentation
  • Context sensitive help for GUI changes
• Testing:
  • Unit tests
  • System (end to end) tests
  • Manual testing
• UX of all users considered:
  • Speech
  • Braille
  • Low Vision
  • Different web browsers
  • Localization in other languages / culture than English
• API is compatible with existing add-ons.
• Security precautions taken.

[CyrilleB79]

Very nice surprise to see this PR! Thanks @RyanMcCleary.

I have not tested yet but have already somme questions.

Given this PR takes into account a location for the subpart being read in the math equation, what about the new Magnifier? Does it follow? In a foreseable future, it would be nice that magnifier can follow the subpart being read, at least in math exploration/navigation mode.If it follows, to which type of tracking is it tied? System focus (which also includes browse mode cursor when applicable)?

[RyanMcCleary]

@CyrilleB79 The magnifier does not yet follow the highlighted subparts. My thinking was that we could handle that in a follow-up PR. As for the type of tracking, system focus seems like the logical choice.

I do still have a couple open questions about this PR where I'd welcome your input. The math highlighter as it's implemented here is a an entirely new highlighter with its own highlighting color (orange). Do you think it should be included in the browse mode highlighter? I can see arguments both for and against this. On the one hand, including it in the browse mode highlighter is simpler and doesn't require adding a new highlighter. On the other hand, math interaction is separate from browse mode, and although it's usually activated from browse mode now, there might be other ways to activate it in the future. Thoughts?

[CyrilleB79]

Yes, Magnifier tracking can be implemented in a follow-up PR. It's even preferable to keep PRs not too big to ease review. I have just mentioned it so that you have it in mind since both features may be related.

My opinion is that browse mode cursor highlighter should be used in this case; the same color should be kept. I agree that this is discutable though.
The reasons why I prefer associating with BM cursor highlighting are:

• when navigating math on the web, there is no longer a browse mode cursor and arrows are actually used to navigate in the math expression
• we have already another situation where browse mode cursor is used when arrows are being used to move something else than the real cursor: navigation in an OCR recognition result. This would lead to a consistent definition of this BM highlighter highlighting locations where arrows are used and where no system caret is actually moved.

[CyrilleB79]

Note, I've tested this PR and highlighting is working well. Thanks!

[RyanMcCleary]

Interesting point re: navigating OCR results. Given that there's already a precedent for this, it does seem like a good idea to use browse mode highlighting. I'll update the PR with that change.

[RyanMcCleary]

Hi @CyrilleB79 Yes, I felt there was still more need for testing on my part before marking it as ready again, and it was my plan to spend some more time going over it last week. I ended up having to catch up on other work though, and I'm hoping to get to it today or tomorrow.

[CyrilleB79]

No problem. Thanks @RyanMcCleary for this update and many thanks again for this work.

[RyanMcCleary]

I've done some more testing using Wikipedia (e.g., the article on the central limit theorem). I've found that some highlights, especially in larger expressions, are appearing in the wrong place. This seems to be due to the fact that Wikipedia renders math visually as an image while exposing the (visually hidden) MathML for screen readers separately.
I'd appreciate any input on how to proceed here. Do we accept this as a limitation of the current highlighting approach? I have not been able to think of any workarounds that would be sufficiently robust, but am of course open to any suggestions.

[CyrilleB79]

Can we detect if the browser uses SVG/Image instead of direct rendering? If yes, we should unfortunately disable highlighting there.

And maybe push against Wikipedia to use native rendering when available.

Alternatively, a browser script modifying the native rendering could probably do the trick but not sure it can be managed by NVDA...

[RyanMcCleary]

@CyrilleB79 This will require a bit of work to get right - I'll spend some time thinking about it and let you know what I come up with. The difficulty is that there are multiple ways the browser can use SVG/images instead of directly rendering MathML, and the DOM and accessibility tree can look different depending on how it's done. It might not be possible to detect in all cases, but we can probably put together a heuristic that works most of the time.

[RyanMcCleary]

After spending a bit more time thinking this over, I'm wondering if we should rethink the general approach. Another approach brought up by @NSoiffer in #19191 is to implement highlighting "by making all navigation to be in a separate window (which I believe is an HTML object)." Even though this seems more disruptive than the approach of this PR, I am wondering if it might be justified by the fact that it avoids the type of difficulties that I discussed earlier in #20372 (comment) where math is rendered as an SVG/image. Using a separate window would allow us to have more control and avoid these types of issues where the highlights are in the wrong place.

[RyanMcCleary]

Marking this as draft again while these higher level questions are still being resolved.

[CyrilleB79]

Maybe we can implement the separate window only as a fallback case if native hilighting is not available ormay be shifted as in svg case.

[RyanMcCleary]

My main concern with that is the unpredictability involved from the user's standpoint. If you imagine you're moving down a page interacting with math, and there's no way to tell before interacting with it whether it will be inline or in a separate window, it seems like that might be quite confusing. That might be acceptable for some users though. Perhaps we could have a setting for "math interaction display" with the following three options:

• Always inline: when an expression can't be accurately highlighted, highlighting is dropped.
• Separate window: always display math interaction in a separate window.
• Inline with separate window fallback: display the interaction with highlights when possible, falling back to highlights in a separate window when necessary.